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ABSTRACT 


A  documented  shortage  of  technical  leadership  and  top- tier  performers  in 
computer  science  jeopardizes  the  technological  edge,  security,  and  economic 
well-being  of  the  nation.  The  2005  President’s  Information  and  Technology 
Advisory  Committee  (PITAC)  Report  on  competitiveness  in  computational 
sciences  highlights  the  major  impact  of  science,  technology,  and  innovation  in 
keeping  America  competitive  in  the  global  marketplace.  It  stresses  the  fact 
that  the  supply  of  science,  technology,  and  engineering  experts  is  at  the  core 
of  America’s  technological  edge,  national  competitiveness  and  security.  How¬ 
ever,  recent  data  shows  that  both  undergraduate  and  postgraduate  production 
of  computer  scientists  is  falling.  The  decline  is  “a  quiet  crisis  building  in  the 
United  States,”  a  crisis  that,  if  allowed  to  continue  unchecked,  could  endanger 
America’s  well-being  and  preeminence  among  the  world’s  nations. 

Past  research  on  expert  performance  has  shown  that  the  cognitive  traits 
of  critical  thinking,  creativity,  and  problem  solving  possessed  by  top-tier  per¬ 
formers  can  be  identified,  observed  and  measured.  The  studies  show  that  the 
identified  attributes  are  applicable  across  many  domains  and  disciplines.  Com¬ 
panies  have  begun  to  realize  that  cognitive  skills  are  important  for  high-level 
performance  and  are  reevahiating  the  traditional  academic  standards  they  have 


used  to  predict  success  for  their  top- tier  performers  in  computer  science. 

Previous  research  in  the  computer  science  field  has  focused  either  on  pro¬ 
gramming  skills  of  its  experts  or  has  attempted  to  predict  the  academic  success 


of  students  at  the  undergraduate  level.  This  study,  on  the  other  hand,  exam- 


°The  views  expressed  in  this  article  are  those  of  the  author  and  do  not  reflect  the  offi¬ 
cial  policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  U.S. 
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ines  the  critical-thinking  skills  found  among  experts  in  the  computer  science 
field  in  order  to  explore  the  questions,  “What  cognitive  skills  do  outstanding 
performers  possess  that  make  them  successful?”  and  “How  do  currently  used 
measures  of  academic  performance  correlate  to  critical-thinking  skills  among 
students?” 

The  results  of  this  study  suggest  a  need  to  examine  how  critical-thinking 
abilities  are  learned  in  the  undergraduate  computer  science  curriculum  and  the 
need  to  foster  these  abilities  in  order  to  produce  the  high-level,  critical-thinking 
professionals  necessary  to  fill  the  growing  need  for  these  experts.  Due  to  the 
fact  that  current  measures  of  academic  performance  do  not  adequately  depict 
students’  cognitive  abilities,  assessment  of  these  skills  must  be  incorporated 
into  existing  curricula. 
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Chapter  1 
Introduction 


The  2005  President’s  Information  and  Technology  Advisory  Committee 
(PITC)  report,  on  competitiveness  in  computer  sciences  highlights  the  major 
impact  of  science,  technology,  and  innovation  in  keeping  America  competitive 
in  the  global  marketplace,  stressing  that  the  supply  of  experts  in  these  fields  is 
at  the  core  of  America’s  technological  edge,  national  competitiveness  and  secu¬ 
rity  [Benioff  and  Lawzoska  2005].  Specifically,  top-tier  performers  in  computer 
science  are  essential  to  the  success  of  business,  according  to  a  2004  Computing 
Research  Association  (CRA)  survey.  Ninety-seven  percent,  of  the  businesses 
surveyed  said  they  could  not  compete  or  even  exist  without  high  performance 
computing  [Vegso  2004].  However,  the  supply  of  these  top  performers  is  de¬ 
clining  at  the  same  time  the  demand  is  growing.  The  impact  of  declining 
numbers  of  experts  in  computer  science  is  “a  quiet  crisis  in  the  United  States” 
that,  if  allowed  to  continue  unchecked,  could  endanger  America’s  well-being 
and  adversely  affect  the  nation’s  current  leadership  role  [Jackson  2005], 

According  to  a  recent  UCLA  Higher  Education  Research  Institute  report, 
the  percentage  of  incoming  undergraduates  at  all  degree- granting  institutions, 
indicating  they  plan  to  major  in  computer  science,  declined  by  70  percent  be¬ 
tween  the  Fall  of  2000  and  2005  [Higher  Education  Research  Institute  2006]. 
The  number  of  new  computer  science  majors  in  the  Fall  of  2005  was  half 
the  number  of  those  in  the  Fall  of  2000  7,952  in  2005  versus  15,958  in  2000. 
A  United  States  Department  of  Education  report  notes  that  other  nations; 
notably  Australia,  China,  India,  Singapore  and  South  Korea;  are  strongly 


Figure  1.1:  Decline  in  computer  science  bachelor’s  degree  production  [Zweben 
2007] 


supporting  degrees  in  the  Science,  Technology,  Engineering,  and  Mathemat¬ 
ics  (STEM)  disciplines,  especially  in  computer  science,  and  investing  heavily 
in  them.  At  the  same  time,  America’s  output  of  these  graduates  is  declin¬ 
ing  [Freeman  2005].  This  marked  decline  in  undergraduate  computer  science 
majors  weakens  the  talent  pool  of  computer  scientists.  Figure  1.1  shows  the 
decline  in  bachelor’s  degree  production  over  the  past  10  years.  According  to 
the  most  recent  Taulbee  CRA  survey,  bachelor’s  production  declined  by  15 
percent  in  2006,  following  a  13  percent  decrease  reported  in  2005  [Zweben 
2007].  One  of  many  contributing  factors  to  this  decline  is  “doubts  about  the 
relevance  of  computing,  particularly  as  it  is  taught’’  [McGettrick  et  al.  2007]. 

The  decreased  enrollment  in  the  United  States  in  computer  science  PhD 
programs  is  equally  as  alarming  as  the  decline  in  undergraduate  enrollment. 
The  number  of  students  entering  computer  science  PhD  programs  in  2005 
decreased  5  percent,  following  an  8  percent  decrease  in  2004  and  a  5  percent 
decrease  in  2003  [Vegso  2006].  According  to  CRA,  the  849  doctoral  degrees 
in  computer  science  and  computer  engineering  awarded  in  2002  by  United 
States  institutions  was  the  lowest  since  1989  [Vegso  2004].  In  2006  the  number 
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Figure  1.2:  Decline  in  production  pipeline  of  PhDs  across  the  United  States 
1984-2006  [Zweben  2007] 


of  new  PhD’s  granted  hit  an  all-time  high  of  1499,  however,  the  long  term 
production  of  PhD's  continues  to  be  a  concern.  The  latest  CRA  report  shows, 
“The  number  of  students  who  passed  the  qualifier  declined  5%,  and  the  total 
number  of  new  Ph.D.  students  declined  more  than  6%  (the  fourth  straight 
year  of  a  decline  in  number  of  new  students)”  [Zweben  2007].  This  overall 
downward  trend  of  PhD  candidates  in  the  production  pipeline  is  shown  in 
Figure  1.2.  This  declined  production  potentially  will  have  a  negative  impact 
on  the  talent  pool  of  computer  science  experts. 

Past  resear  cl  i  lias  identified  specific  traits  and  cognitive  skills  common 
among  the  top-tier  performers  or  experts  in  their  fields.  However,  traditional 
methods  of  identifying  potential  experts  often  rely  solely  on  assessing  subject- 
matter  expertise  and  evaluating  measures  such  as  academic  performance  and 
technical  skills.  Some  companies,  however,  have  found  that  these  methods  do 
not  necessarily  predict  success.  For  example,  Intel  and  Google  have  found  that 
successful  and  innovative  employees  did  not  necessarily  have  high  grades  in  un¬ 
dergraduate  computing  courses,  did  not  uniquely  come  from  elite  engineering 
institutions,  and  do  not  always  have  the  widest  array  of  programming  skills  and 
technical  expertise  [Colwell  2005].  On  the  other  hand,  more  recent  research. 
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Figure  1.3:  Study  direction  for  critical- thinking  skills  in  computer  science  as 
proposed  in  the  research  presented  here 

using  other  methods  of  identification,  has  shown  that  expert  performers  have 
strengths  in  the  cognitive  skills  of  problem  solving,  critical  thinking  and  cre¬ 
ativity  [Sternberg  2003b;  2006;  Ericsson  2003;  Facione  2005;  Chi  and  Glaser 
1988]. 

These  cognitive  skills  also  have  great  significance  for  expert  performance 
in  computer  science.  Traditional  computer  science  curriculum  and  undergrad¬ 
uate  education  focus  on  subject-matter  knowledge  as  defined  by  The  Joint 
Task  Force  oil  Computing  Curricula  [ACM  and  IEEE  2001;  1991],  Recent 
guideline  recommendations  by  the  Accreditation  Board  for  Engineering  Tech¬ 
nology  (ABET)  Computing  Accreditation  Committee  (CAC),  however,  have 
expanded  the  basic  skill  requirements,  placing  a  new  emphasis  on  critical- 
thinking  and  problem-solving  skills.  It  requires  accredited  programs  to  equip 
graduates  with  “an  ability  to  analyze  a  problem,  and  identify  and  define  the 
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computing  requirements  appropriate  to  its  solution"  and  “an  ability  to  design, 
implement  and  evaluate  a  computer-based  system,  process,  component,  or  pro¬ 
gram  to  meet  desired  needs”  [ABET  2007].  A  recent  report  by  the  Computer 
Science  Teachers  Association  has  likewise  stressed  the  importance  of  critical- 
thinking  and  problem-solving  skills,  listing  among  its  10  core  principles  of 
computer  science  education  a  “focus  on  teaching  problem-solving  methodolo¬ 
gies  and  critical-thinking  skills,”  and  a  need  to  “help  students  develop  a  wide 
range  of  cognitive  capabilities  and  practical  skills,  independent  of  specific  tech¬ 
nologies”  [CSTA  Curriculum  Improvement  Task  Force  2005]. 

Current  studies  have  begun  to  evaluate  the  importance  of  critical- thinking 
skills  in  computer  science  and  how  they  should  be  incorporated  into  specific 
core  courses  [M.R.K.  Krishna  Rao  and  Bagais  2006;  Fagin  et  al.  2006];  however, 
a  thorough  review  of  these  studies  and  of  available  computer  science  literature 
show  that  research  into  critical-thinking  skills  among  computer  professionals 
and  experts  and  how  to  relate  these  skills  to  undergraduate  education  is  lack¬ 
ing.  One  of  the  first  steps  in  addressing  this  needed  research  is  to  compare  and 
contrast  the  critical-thinking  skills  of  top-tier  professionals  and  undergradu¬ 
ate  students,  and  to  assess  correlations  between  currently  used  measures  of 
academic  success  and  these  skills,  raising  several  questions  reported  in  this 
study. 

The  first  research  question  examines  top-tier  performers  in  computer  sci¬ 
ence  in  an  effort  to  identify  the  critical-thinking  skills  that  have  enabled  them 
to  become  the  experts.  The  study  also  examined  college  freshman  and  se¬ 
nior  computer  science  students  in  order  to  answer  additional  research  ques¬ 
tions:  “What  critical-thinking  abilities  do  freshmen  have?”;  “What  are  the 
critical-thinking  abilities  of  senior  computer  science  students?”;  “How  do  the 
critical-thinking  skills  of  the  freshmen  and  senior  computer  science  students 
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Figure  1.4:  Educational  impac  t,  of  cognitive  skills  on  expert  performance 

compare?’1,  and  “How  do  currently  used  measures  of  academic  performance, 
such  as  grade- point  average,  correlate  to  this  assessment  of  critical-thinking 
skills?”  The  results  of  these  questions  enabled  the  research  to  answer  the  final 
question,  “What  are  the  differences  in  critical-thinking  skills  between  profes¬ 
sionals  and  undergraduate  computer  science  students?"  Figure  1.4  graphically 
illustrates  the  role  that  educating  college  students  in  cognitive  skills  plays  in 
equipping  them  to  become  those  export  top-tier  performers  in  their  fields. 
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While  traditional  measures  of  academic  success  are  important  and  underlying 
subject-specific  skills  are  at  the  core  of  a  computer  science  major,  this  study 
shows  that  critical- thinking  skills  do  have  implications  for  computer  science 
education.  It  suggests  a  need  to  reevaluate  the  computer  science  curriculum 
in  an  effort  to  address  the  concerns  caused  by  the  growing  demand  for  top-tier 
experts  in  the  field. 

Chapter  1  has  included  information  related  to  the  problem  that  prompted 
this  research,  a  statement  of  that,  problem,  and  the  significance  of  this  study. 
Chapter  2  provides  a  foundation  for  this  research  by  reviewing  the  background 
literature  related  to  expertise,  problem  solving,  critical  thinking  and  creativity. 
Chapter  3  explains  the  research  methodology  used,  and  Chapter  4  analyzes  the 
results.  Chapter  5  concludes  the  dissertation  with  a  summary  of  the  study, 
its  implications  for  computer  science  education,  and  suggested  areas  for  future 
work. 
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Chapter  2 
Background 


2.1  Theories  of  Expert  Performance 

The  study  of  expert  performance,  a  widely  researched  subject  in  cognitive 
psychology,  has  found  that  expert  performers  possess  certain  common  traits: 
a  mastery  of  the  subject  content  in  which  they  are  involved,  and  high-level 
strengths  in  3  basic  cognitive  areas-problem  solving,  critical  thinking  and  cre¬ 
ativity.  Figure  2.1  synthesizes  these  common  traits  as  reported  by  top  re¬ 
searchers: 


Expert  Performer 


Cognitive 

Abilities 


Figure  2.1:  Factors  that  influence  expert  performance 


Gallon  (1822-1911),  one  of  the  earliest  pioneers  in  the  study  of  traits  of 
expert  performers,  stands  out  because  he  was  the  first  to  theorize  and  explore 


factors  of  expert  performance,  including  an  innate  ability,  eagerness  to  work, 
and  “an  adequate  power  of  doing  a  great  deal  of  very  laborious  work”  [Ericsson 
and  Charness  1994].  Although  he  considered  the  later  2  important,  his  research 
focused  primarily  on  innate  abilities  and  the  roie  heredity  plays  in  intelligence 
and  expert  performance.  He  t  heorized  that  innate  abilities  are  necessary  for 
excellence  and  that  although  practice  is  required,  it  alone  cannot  produce  top 
performers.  In  Hereditary  Genivs  (1869),  he  espouses  his  views  that  hereditary 
factors  form  the  upper  limit  to  maximal  performance.  He  uses  the  example  of 
beginning  a  new  physical  activity  such  as  rowing,  lifting  weights,  or  running: 

So  long  as  he  is  a  novice,  he  perhaps  Hatters  himself  there  is 
hardly  an  assignable  limit  to  the  education  of  his  muscles;  but  the 
daily  gain  is  soon  discovered  to  diminish,  and  at  last  it  vanishes 
altogether.  His  maximum  performance  becomes  a  rigidly  determi¬ 
nate  quantity _ There  is  a  deHnite  limit  to  the  muscular  powers  of 

every  man,  which  he  cannot  by  any  education  or  exertion  overpass. 

In  extending  these  physical  limits  to  the  intellectual  domain,  he  argues  that 
“this  is  precisely  analogous  to  the  experience  that  every  student  has  had  of  the 

working  of  his  mental  powers _ ”  As  the  student  matures,  he  reaches  naturally 

inherited  limits  to  his  intellectual  capacity  and  “when  he  reaches  mature  life, 

he  is  confident  only  within  certain  limits - ”  He  “limits  his  undertakings  to 

matters  below  the  level  of  his  reach  and  finds  true  moral  repose  in  an  honest 
conviction  that  he  is  engaged  in  as  much  good  work  as  his  nature  has  rendered 
him  capable  of  performing”  [Galton  1892], 

Theorizing  that  excellence  in  diverse  Helds  and  domains  has  a  common 
set  of  causes,  Galton  studied  such  characteristics  as  “height,  body  size,  cir¬ 
cumference  of  head,  size  of  brain,  weight  of  grey  matter  and  number  of  brain 
Hbers"  in  ati  attempt  to  Hnd  physiological  differences  between  experts  and 
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novices  [Simonton  2003].  His  views  of  “maximum  performance”  based  on  ge¬ 
netic  qualities,  shaped  much  of  the  early  debate  and  research  on  expertise  and 
intelligence.  “Galton’s  belief  in  the  adaptive  value  of  natural  ability  became 
thereby  translated  into  widespread  conviction  that  general  intelligence  pro¬ 
vides  the  single  most  critical  psychological  factor  underlying  success  in  life” 
[Simonton  2003].  His  study,  “On  Men  of  Science”  (1874),  in  which  he  looks 
at  both  innate  and  environmental  factors  that  shape  excellence,  further  doc¬ 
uments  his  belief  and  is  particularly  famous  because  it  introduces  the  phrase, 
“nature  and  nurture,”  to  describe  interacting  factors  that  promote  high  levels 
of  human  achievement  or  expertise.  In  Galton’s  words,  nature  is  “all  that  a 
man  brings  into  the  world;  nurture  is  every  influence  without  that  affects  him 
since  his  birth”  [Galton  1874],  Leading  researchers  continue  to  acknowledge 
the  impact  of  his  work  on  their  studies. 

Spearman  (1863-1945),  another  researcher  who  recognized  innate  ability 
as  being  of  primary  importance  in  the  development  of  an  expert  performer, 
“invented  factor  analysis,  a  method  which  permitted  a  rigorous  statistical  test 
of. . .  Gallon's  hypothesis  that  a  general  mental  ability  enters  into  every  kind 
of  activity  requiring  mental  effort”  [Jensen  1999].  His  g-theory,  or  the  two- 
factor  theory,  divides  expertise  into  two  areas,  the  general  or  g  factor,  and  the 
area  more  specific  to  a  discipline,  the  s  factor.  For  him,  the  general  factor  of 
“mental  energy'”  determines  individual  differences  in  expertise.  To  make  this 
relationship  more  quantifiable,  Spearman  developed  a  formula  that  combined 
general  ability  or  intelligence  (the  g  factor)  with  the  specific  factor  (the  s  fac¬ 
tor)  in  a  particular  venue  to  predict  whether  or  not  an  individual  can  be  an 
expert  [Spearman  1904],  His  findings  that  intelligent  behavior  arises  from  a 
“single  metaphorical  entity,”  forms  the  foundation  for  many  theories  of  human 
intelligence  [Jenson  1998].  His  factor  theory  is  considered  by  some  to  be  “the 
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most  important  psychometric  construct  in  the  study  of  individual  differences 

in  human  cognitive  abilities _ The  g  factor  has  become  so  firmly  established 

as  a  major  psychological  construct  in  terms  of  psychometric  and  factor  ana¬ 
lytic  criteria  that  further  research  along  these  lines  is  very  unlikely  either  to 
disconfirm  the  construct  validity  of  g  or  to  add  anything  essentially  new  to  our 
understanding  of  it”  [Jensen  1999]. 

In  contrast  to  Galton’s  theories  of  innate  abilities,  Binet’s  (1857-1911)  sem¬ 
inal  research  at  the  turn  of  the  twentieth  century  focused  on  developing  an  as¬ 
sessment  to  support  his  hypothesis  that  environmental  factors  play  a  larger  role 
in  intelligence  and  expertise  than  inherited  abilities  do.  Binet’s  methods  of  di¬ 
agnosing  levels  of  intellect- the  medical  method  (anatomical,  physiological,  and 
pathological  signs  of  inferior  intelligence),  the  pedagogical  method  (judging 
intelligence  according  to  the  sum  of  acquired  knowledge),  and  the  psychologi¬ 
cal  method  (observations  and  measurements  of  the  degree  of  intelligence)  led 
to  his  development  of  several  tests,  including  the  Stanford- Binet  intelligence 
test  [Binet  1905]  that  assesses  the  natural  intellectual  abilities  of  individuals. 
Many  of  the  current  theories  of  intelligence  arrive  from  the  contributions  of 
and  theoretical  differences  between  Galton  and  Binet  [Sternberg  2003a].  Binet 
expanded  his  research  with  extensive  studies  of  the  cognitive  abilities  of  chess 
experts  laying  the  foundation  for  the  studies  of  de  Groot  and  others  in  the 
1950’s.  He  originally  thought  chess  masters  had  superior  memory  and  recall, 
but  concluded  that  experience,  imagination  and  memory  play  a  large  role  in 
the  level  of  expertise  required  in  grand  master  chess  [Binet  1894], 

Whereas  Binet  and  Spearman  emphasized  the  innate  abilities  of  expert 
performers,  Poincare  (1854-1912)  was  one  of  the  first  researchers  to  emphasize 
the  creative  portion  of  the  problem-solving  and  critical-thinking  processes. 
His  study  of  the  role  of  creativity  in  solving  problems  laid  the  groundwork 
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for  his  theory  of  the  generation  of  creative  ideas.  The  “appearance  of  sudden 
illumination  is  a  manifest  sign  of  long,  unconscious  prior  work.”  Initial,  intense, 
prior,  conscious  work  on  a  problem  is  necessary  to  “unlock”  relevant  ideas  from 
their  “fixed  positions  so  they  are  free  to  join  during  unconscious  processing” 
[Poincare  1907] .  This  process  is  initiated  by  conscious  but  unsuccessful  efforts 
to  solve  a  problem,  followed  by  the  unconscious  phase  that  ultimately  leads 
to  a  collection  of  potential  solutions  from  which  one  solution  emerges.  On 
the  other  hand,  to  doubt  everything  or  to  believe  everything  are  two  equally 
convenient  solutions;  both  ignore  the  necessity  of  reflection,  the  process  that 
leads  to  the  “Aha!”  moment  [Poincare  1907]. 

Following  the  research  and  writings  of  Kant  (1724-1804),  Dedekind  (1831- 
1916),  and  Brouwer  (1881-1966),  Poincare  considered  mathematics  as  intu¬ 
ition,  not  pure  logic.  “It  is  by  logic  that  we  prove,  but  by  intuition  that  we 
discover”  [Poincare  1905].  According  to  him,  intuition  is  what  “goes  on  in  the 
very  soul  of  the  mathematician.”  In  his  work.  The  Foundations  of  Science ,  in 
which  he  relates  his  struggle  to  explain  the  mathematical  Fuchsian  function, 
he  describes  the  creative  component  of  the  problem-solving  process,  positing 
that  unconscious  thought  offers  a  “point  of  depart  me”  from  which  the  con¬ 
scious  mind  can  work  out  the  argument  in  detail.  The  conscious  mind,  on 
the  other  hand,  is  capable  of  the  strict  discipline  and  logical  thinking  of  which 
the  unconscious  Ls  incapable  [Poincare  1908].  His  theories  added  new  insight 
to  the  scientific  method  described  by  Dewey  laying  the  foundation  for  future 
studies  in  creativity  and  its  role  in  the  critical-thinking  and  problem-solving 
processes. 

Dewey  (1859-1952),  agreeing  with  Poincare,  theorized  that  the  problem¬ 
solving  process  involves  both  logic  he  termed  “reflection”,  and  intuition  which 
he  saw  as  “unconscious  thought”.  Whereas  Poincare  emphasis  was  on  a  more 
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creative  method  of  solving  problems,  Dewey  saw  the  process  from  a  more  struc¬ 
tured  and  classical  “scientific  method”  [Deek  et  al.  1999].  According  to  Dewey, 
a  critical  thinker  and  problem  solver  must  develop  the  habit  of  reflect  ive  think¬ 
ing,  as  it  forms  the  basis  for  critical  inquiry;  it  leads  somewhere,  to  a  specific 
albeit  initially  unknown  goal  or  conclusion;  it  is  the  ability  to  suspend  judg¬ 
ment,  to  maintain  a  healthy  skepticism,  and  to  exercise  an  open  mind.  It  is 
an  “active,  persistent,  and  careful  consideration  of  any  belief  or  supposed  form 
of  knowledge  in  the  light  of  the  grounds  that  support  it  and  the  further  con¬ 
clusions  to  which  it  tends"  [Dewey  1910].  As  this  definition  suggests,  reflective 
and  critical  thought  have  both  an  intellectual  and  an  emotional  component. 

Although  best  known  as  a  philosopher  and  a  pragmatist,  Dewey  made 
several  seminal  contributions  as  an  educational  theorist.  The  central  con¬ 
cept  of  his  view  of  education  is  that  greater  emphasis  should  be  placed  on 
the  broadening  of  the  intellect  and  the  development  of  problem-solving  and 
critical -thinking  skills  rather  than  on  the  memorization  of  lessons,  a  popular 
educational  method  of  the  time.  Put  simply,  his  theory  promotes  “learn  by 
doing”  rather  than  through  practice  and  repetition  [Dewey  1916].  His  problem¬ 
solving  step-by-step  process  (“active  learning”),  well  known  and  often  quoted, 
consists  of  the  following  steps: 

•  Definition  and  analysis  of  the  problem 

•  Establishment  of  criteria  for  evaluating  solutions 

•  Identification  of  possible  solutions 

•  Selection  of  the  best,  solution 

•  Testing  of  the  selected  solutions 
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Dewey’s  research  into  reflective  thought;  the  basis  for  many  theories  of 
problem  solving,  critical  thinking  and  creativity;  influenced  significantly  the 
later  works  of  Guilford  (1950),  Polya  (1957),  and  Sternberg  (1996,  1998,  2003). 
His  book,  How  We  Think  published  in  1910  and  revised  in  1933  [Dewey  1910: 
1933],  establishes  the  framework  for  the  study  of  top-tier  performers  in  com¬ 
puter  science  reported  in  this  dissertation. 

The  French  mathematician,  Hadamard  (1865-1963),  probably  best  known 
for  proving  the  Prime  Number  Theorem,  also  drew  upon  Poincare’s  problem¬ 
solving  theories  in  his  work.  The  Psychology  of  Invention  in  the  Mathematical 
Field  (1954).  In  it  he  discusses  the  creative  process  in  the  discovery  segment  of 
the  problem-solving  cycle,  the  unique  process  used  by  mathematicians,  physi¬ 
cists  and  engineers.  He  observed  that,  this  creative  process  is  composed  largely 
of  wordless  mental  images  that  form  the  entire  solution  set  to  the  problem 
at  hand  [OConnor  and  Robertson  2003].  His  four-step  model  (preparation, 
incubation,  illumination  and  verification)  stresses  the  importance  of  insights 
derived  from  the  incubation  portion  of  problem  solving  [Hadamard  1954], 

Guilford  (1897-1987)  expanded  the  earlier  work  of  Dewey  and  recognized 
fundamental  traits  of  creativity.  In  his  1950  address  to  the  American  Psy¬ 
chological  Association,  he  revitalized  modern  research  into  creativity  and  di¬ 
vergent  thinking  and  emphasized  the  difference  between  convergent  thinking, 
deriving  the  single  best,  answer  to  a  clearly  defined  problem,  and  divergent 
thinking,  producing  multiple  or  alternate  answers  from  available  information 
[Guilford  1950].  His  theories  emphasize  the  belief  that  the  first  step  in  the 
problem-solving  process,  analyzing  the  problem,  requires  creativity  and  diver¬ 
gent  tl linking  to  establish  a  series  of  possible  solutions. 

He  also  rejected  the  earlier  notions  of  a  two-factor  theory  of  intelligence  as 
proposed  by  Spearman.  In  a  study  of  highly  skilled  personnel,  he  expresses 
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his  belief  that  “thinking  abilities,  which  have  played  important  roles  in  some 
definitions  of  intelligence,  seemed  to  have  been  neglected;  particularly  abil¬ 
ities  having  to  do  with  productive  thinking”  [Guilford  1956].  He  believed 
that  a  general  test,  such  as  the  Binet- Stanford  Intelligence  Test  with  a  sin¬ 
gle  score,  can  measure  the  variance  of  intelligence  and  expertise  in  only  one 
or  two  factors.  “Assessment  of  intellectual  qualities  should  go  much  beyond 
present  standard  intelligence  tests  which  seriously  neglect  important  abilities 
that  contribute  to  problem  solving  and  creative  performance  in  general”  [Guil¬ 
ford  1968]. 

Guilford  also  popularized  the  structure  of  intelligence  (SI)  model,  depicted 
in  Figure  2.2.  It  provides  for  120  different  SI  abilities  that  factor  into  in¬ 
telligence  and  expert  performance  and  includes  components  of  divergent  and 
convergent  creativity  [Kearsley  2004].  In  it  the  interplay  between  convergent 
and  divergent  thinking  is  a  key  part  of  intellect. 
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Figure  2.2:  Structure  of  Intelligence  model  proposed  by  Guilford  [Kearsley 
2004] 
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He  concluded  that  previous  attempts  to  measure  “the  superior  human 
adult,”  or  expert,  had  a  far  too  narrow  scope  and  believed  that  “psychology 
and  psychologists  since  Binet  have  taken  a  much  too  restricted  view  of  human 

intelligence _ In  attempting  to  fathom  the  nature  of  intellect,  more  attention 

should  be  given  to  the  human  adult,  particularly  the  superior  human  adult. 
It  is  to  such  specimens  that  we  must  go,  if  we  are  to  investigate  intellectual 
qualities  and  functions  in  their  greatest  scope  and  variety”  [Guilford  1950]. 
The  consideration  of  divergent  production  is  “anything  but  a  minor  innova¬ 
tion.  Guilford  brought  within  the  realms  of  a  problem  not  just  a  new  idea,  but 
the  rest  of  t  he  universe-or  whatever  part  of  it  might  be  helpful  at  the  time  in 
finding  a  creative  solution,  be  this  from  the  past,  present,  or  future”  [Richards 
2001], 

Building  on  the  work  of  Dewey  and  Poincare,  Polya  (1887-1985)  devised 
steps  in  the  problem-solving  cycle  [Polya  1957]: 

•  Understand  the  problem 

•  Find  the  connection  between  the  data  and  the  unknown 

•  Devise  a  plan  and  take  action  on  a  solution 

•  Examine  the  results  obtained 

His  classic  work  on  problem  solving,  How  to  Solve  It,  gives  genera]  heuristics 
for  solving  problems  of  all  kinds  and  across  all  disciplines  [Polya  1957].  In  it  he 
provides  “a  list  of  heuristics  for  understanding  a  problem  and  devising  a  plan 
to  solve  it,  including  making  sure  that  the  givens,  the  conditions,  and  the  goal 
stated  are  understood;  reformulating  the  problem;  thinking  of  known  analo¬ 
gous  problems;  making  the  problem  more  general;  and  breaking  the  problem 
into  parts”  [Frederiksen  1984].  This  seminal  work  has  had  a  major  impact  on 
the  theory  of  problem  solving  across  STEM  disciplines. 
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Iii  Mathematics  and  Plausible  Reasoning,  Volume  I.  Polya  discusses  induc¬ 
tive  reasoning,  reasoning  from  particular  cases  to  the  general  rule  in  math¬ 
ematics.  In  Volume  II,  he  gives  equal  expression  to  his  interests  in  mathe¬ 
matics,  natural  science  and  cognitive  psychology  [Polya  1954],  and  discusses 
more  general  forms  of  inductive  logic.  One  of  his  central  themes  stresses  the 
role  of  mathematical  engagement,  the  active  engagement  of  discovery,  one 
that  takes  place  in  large  measure  by  guessing.  He  argues  that  mathemat¬ 
ics  is  not  an  entirely  formal  deductive  discipline,  but,  like  the  sciences,  it  is 
an  inductive  discipline  that  requires  conjecture,  insight  and  discovery.  His 
problem-solving  theories  are  cited  in  journals  such  as  American  Political  Sci¬ 
ence  Review,  Annual  Review  of  Psychology,  Artificial  Intelligence,  Computers 
and  Chemistry,  Computers  and  Education.  Discourse  Processes,  Educational 
Leadership,  Higher  Education,  and  Human  Learning  [Schoenfeld  1992]. 

Building  upon  Bi net’s  influence  and  his  belief  that  human  intelligence  can 
be  assessed,  Piaget  (1896-1980)  developed  a  four-stage  model  of  human  intel¬ 
lectual  development  for  use  in  classifying  intellectual  abilities.  He  sought  to 
explain  the  stages  of  intellectual  development,  the  top  stage  which  he  called 
“formal  operational  reasoning,”  and  defined  as  “the  ability  to  use  abstract  rea¬ 
soning  and  deduction  as  well  as  to  employ  previous  knowledge  and  experiences 
to  less  well-defined  situations”  [Piaget  1972].  The  characteristics  of  this  stage 
of  development  axe  ’’exhibited  in  an  individual’s  ability  to  carry  out  combi¬ 
natorial  analysis,  propositional  logic. . . ,  proportional  reasoning,  and  isolating 
and  controlling  relevant  variables  from  among  the  set  of  identified  possibilities 
the  individual  has  generated”  [Nurrenbern  2001],  Piaget  also  has  had  a  con¬ 
siderable  impact  on  the  field  of  artificial  intelligence.  Seymour  Papert,  who 
saw  Piaget  as  a  “giant  in  the  field  of  cognitive  theory,”  [Papert  1999]  used  his 
work  while  developing  the  Logo  programming  language. 
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Although  other  researchers  had  developed  their  models  for  learning,  Bloom 
(1913-1999)  and  his  colleagues  at  Chicago  University  saw  the  need  to  obtain 
more  reliable  evaluations  and  to  provide  improved  procedures  in  education. 
They  were  “interested  in  providing  a  useful  practical  tool  that  was  congruent, 
with  what  was  understood  at  that  time  about  the  features  of  the  higher  men¬ 
tal  processes”  [Eisner  2000]  and  devised  a  system  that,  consists  of  6  levels  of 
learning,  each  one  building  on  the  level  below  it  and  increasing  in  complexity 
[Bloom  1956].  The  following  is  Bloom’s  definition  of  each  level  with  sample 
learning  objectives  [Nilson  2003]: 

•  Knowledge  is  the  recall  of  information  previously  learned,  the  foundation 
for  the  higher  levels  of  thinking.  “The  student  will  be  able  to  state 
Newton’s  Laws  of  Motion  ” 

•  Comprehension  is  the  ability  to  understand  the  meaning  of  what  has 
been  learned  and  the  ability  to  interpret  and  explain  it.  The  student  will 
be  able  to  describe  the  trends  in  the  graph  in  her  own  words. 

•  Application  is  the  ability  to  apply  what  has  been  learned  in  different 
situations.  “The  student  will  be  able  to  determine  the  variables  to  be 
controlled  for  an  experiment.” 

•  Analysis  is  the  ability  to  separate  learned  material  into  component  parts 
and  to  show  the  relationships  between  those  parts.  “The  student  will 
be  able  to  describe  an  experiment  to  test  the  influence  of  light  and  light 
quality  on  the  Hill  reaction  of  photosynthesis.” 

•  Synthesis  is  the  ability  to  put  separate  ideas  or  learned  facts  together 
to  form  them  into  new  relationships  and  forms.  “The  student  will  be 
able  to  compose  a  logical  argument  on  assisted  suicide  in  opposition  to 
personal  opinion.” 
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•  Evaluation  is  the  ability  to  assess  the  available  information  and  to  make 
appropriate  judgments  about  it.  “The  student  will  be  able  to  assess  the 
validity  of  certain  conclusions  based  on  the  data  and  statistical  analysis.” 

Better  known  as  “Bloom’s  Taxonomy,"  his  model  of  the  stages  of  learning  in  the 
cognitive  domain  is  widely  used,  largely  because  it  can  be  understood  easily. 

The  top  3  levels -analysis,  synthesis  and  evaluation  directly  involve  critical 
thinking  and  are  characteristic  of  experts  in  all  domains.  All  of  the  6  levels, 
however,  can  apply  to  computer  science  education  in  general.  The  following 
sample  assignments  illustrate  that  fact  [Scott  2003] : 

•  Knowledge;  “Name  the  three  kinds  of  looping  structures  in  C++.... 
State  5  things  that  are  true  of  a  RISC  architecture,” 

•  Comprehension:  “Indicate  why  more  registers  inside  the  CPU  can  make 
the  processor  faster.” 

•  Application:  “Demonstrate  different  programming  constructs  in  1  assign¬ 
ment.” 

•  Analysis:  “Compare  RISC  and  CISC  architectures.” 

•  Synthesis:  “Design  samples  of  inheritance  or  polymorphism.” 

•  Evaluation:  “Organize  a  complete  test  plan  for  a  programming  assign¬ 
ment.” 

The  theories  of  de  Groot  (1914-2006)  also  grew  from  and  built  upon  those 
of  earlier  researchers.  He  applied  Binet’s  assessment  of  intelligence  anti  Pi¬ 
aget’s  model  of  learning  to  examine  short-term  recall  and  expertise  of  master 
chess  players.  In  his  groundbreaking  study,  he  demonstrated  the  possibility 
of  directly  studying,  under  controlled  laboratory  conditions,  the  thought  pro 
cesses  mediating  the  highest  levels  of  performance  of  expert  and  master  chess 
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players.  He  instructed  them  to  think  aloud  as  they  selected  their  next  move  for 
a  series  of  chess  positions  taken  from  unfamiliar  games  by  chess  masters.  Fun¬ 
damentally,  he  showed  that  world-class  players  select  better  moves  for  chess 
positions  than  skilled  club  players  do  and  provided  initial  insights  into  the 
processes  mediating  the  difference  [Ross  2006].  “We  know  that  increasing  ex¬ 
perience  and  knowledge  in  a  specific  field  (chess,  for  instance)  has  the  effect 
that  things  (properties,  for  example)  which,  at  earlier  stages,  had  to  be  ab¬ 
stracted,  or  even  inferred  are  apt  to  be  immediately  perceived  at  later  stages. 
To  a  rather  large  extent,  abstraction  is  replaced  by  perception —  A  so-called 
'given’  problem  situation  is  not  really  given  since  it  is  seen  differently  by  an 
expert  than  it  is  perceived  by  an  inexperienced  person...”  [de  Groot  1965]. 
His  findings  on  the  structure  of  chess  skill  led  to  one  of  the  first  theories  of 
expertise  of  chess  players  [Ericsson  2005]. 

Colleagues  Chase  and  Simon  (1973)  have  expanded  on  de  G root’s  work  with 
chess  players  by  proposing  a  10- year  rule  for  the  development  of  chess  expertise. 
They  believe  that  one  needs  4  hours  of  study  a  day  for  approximately  a  decade 
in  order  to  acquire  the  necessary  knowledge  base  for  performing  at  high  levels 
in  any  domain.  They  propose  that  with  extended  experience,  experts  acquire  a 
larger  number  of  increasingly  complex  patterns  of  chess  pieces  (chunks)  and  use 
them  to  retrieve  moves  (actions)  when  similar  chess  positions  are  encountered 
during  subsequent  chess  playing  [Ericsson  2005]. 

An  important  areas  of  research  of  cognitive  abilities  is  the  modeling  of  hu¬ 
man  expertise  with  computer  systems  [Feitovich  et  al.  2006],  Research  in  this 
area  is  a  growing  field  including  the  work  of  Lenant  on  large  knowledge-based 
models  [Lenant  and  Guha  1990],  the  work  of  Mitchell  oil  knowledge- based 
learning  [Mitchell  1997],  and  the  knowledge  extraction  work  of  Hammond  and 
Davis  [Hammond  and  Davis  2004].  Various  researchers  pioneered  work  in  the 
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area  including  Miller  (I960),  R.eitman  (1965),  and  Newell  (1956,  1973)  with 
their  early  works  on  artificial  intelligence  (AI)  [Feltovich  et  al.  2006]  that  at¬ 
tempts  to  model  intelligence  by  building  computer  programs  able  to  exhibit 
intelligent  behavior  [Buchanan  et  al.  2006].  Turing  considered  the  question, 
“Can  machines  think?"  in  his  seminal  work,  “Computing  Machinery  and  Intel¬ 
ligence.” 

In  about  fifty  years  time  it  will  be  possible  to  program  com¬ 
puters.  . .  to  make  them  play  the  imitation  game  so  well  that  an 
average  interrogator  will  not  have  more  than  70  per  cent  chance 
of  making  the  right,  identification  after  five  minutes  of  questioning. 

The  original  question,  ‘Can  machines  think?’  i  believe  to  be  too 
meaningless  to  deserve  discussion.  Nevertheless,  1  believe  that  at. 
the  end  of  the  century  (twentieth)  the  use  of  words  anti  general 
educated  opinion  will  have  altered  so  much  that  one  will  be  able  to 
speak  of  machines  thinking  without  expecting  to  be  contradicted 
[Turing  1950]. 

Research  into  developing  systems  that  mimic  expert  behavior  have  sparked  new 
questions  into  the  study  of  expertise,  such  as,  “Can  expert- level  performance 
be  achieved  by  a  computer  program?";  “How  can  tacit  knowledge  be  made 
explicit?”,  and  “Are  some  types  of  knowledge  more  critical  to  high  performance 
than  others?"  [Feltovich  et  al.  2006].  Attempts  to  model  expert  behavior  and 
the  problem-solving  cycle  by  means  of  computer  expert  systems  influenced  the 
development  of  current  theories  dealing  with  cognitive  abilities. 

One  of  the  most  important  initiatives  by  the  early  researchers  of  expert  sys¬ 
tems  was  the  development  of  the  “General  Problem  Solver"  (GPS)  by  Newell 
and  Simon,  based  on  their  earlier  work,  “Logic  Theory  Machine”  (1956).  The 
GPS  used  a  “means-ends  analysis,”  describing  the  desired  goal  state,  looking 
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at  the  present  place  in  the  problem-solving  process,  selecting  a  transition  to  a 
new  state  and,  at  the  same  time,  remembering  the  final  goal  [Gardner  1985]. 
Newell  stressed  the  resemblance  between  human  and  macliine  problem  solving 
[Gardner  1985].  He  believed  that  all  intelligence  involves  the  use  and  ma¬ 
nipulation  of  various  symbols  and  that  “the  profound  similarities  between  the 
human  mind  engaged  in  solving  a  problem,  and  the  computer  programmed  to 
solve  the  same  problem,  far  overrode  differences  in  hardware —  Both  are  sim¬ 
ply  systems  that  process  information  over  time. . [Gardner  1985].  In  Hum, an 
Problem  Solving ,  he  outlines  the  steps  he  believes  necessary  for  a  computer  to 
model  human  thought  [Newell  1971]: 

•  “Discover  and  define  a  set  of  processes. . .  of  storing  and  manipulating 
patterns  to  perform  complex  nonnumerical  tasks." 

•  “Construct  an  information- processing  language,  and  a  system  for  inter¬ 
preting  that  language  in  terms  of  elementary  operations.” 

•  “Discover  and  define  a  program. . .  capable  of  solving  some  class  of  prob¬ 
lems  that  humans  find  difficult.” 

•  “Obtain  data, . .  on  human  behavior  in  solving  the  same  problems  as  those 
tackled  by  the  program.  Search  for  the  similarities  and  differences  be¬ 
tween  the  behavior  of  program  and  human  subject.  Modify  the  program 
to  achieve  a  better  approximation  to  the  human  behavior.” 

•  “Investigate  a  continually  broadening  range  of  human  problem-solving 
and  thinking  tasks." 

•  “Construct  more  general  simulation  programs  that  can  attack  a  whole 
range  of  tasks." 
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•  “Examine  the  components  of  the  simulation  programs  for  their  relation 
to  the  more  elementary  human  performances  that  are  commonly  stud¬ 
ied  in  the  psychological  laboratory:  rote  learning,  elementary  concept 
attainment,  immediate  recall,  and  so  on.  Draw  inferences  from  simula¬ 
tions  to  elementary  performances,  and  vice  versa,  so  as  to  use  standard 
experimental  data  to  test  and  improve  the  problem-solving  theories.” 

•  “Search  for  new  tasks. . .  that  might  provide  additional  arenas  for  testing 
the  theories.” 

•  “Use  neurophysiological  evidence  to  improve  the  problem-solving  theo¬ 
ries.” 

•  “Draw  implications  from  the  theories  for  the  improvement  of  human  per¬ 
formance,  for  example,  the  improvement  of  learning  and  decision  mak¬ 
ing.” 

•  "Review  progress.” 

Other  innovations  in  artificial  intelligence  and  expert  systems  have  had 
equally  profound  affects  on  cognitive  research  and  the  study  of  expert  perfor¬ 
mance.  John  McCarthy  at  M.I.T.  (1962)  developed  a  list,- processing  program¬ 
ming  language,  LISP,  to  mimic  the  “mental  steps  of  problem  solving —  He 
believed  the  route  to  making  machines  intelligent  is  through  a  rigorous  formal 
approach  in  which  the  acts  that  make  up  intelligence  are  reduced  to  a  set  of 
logical  relationships  or  axioms  that  can  be  expressed  precisely  in  mathematical 
terms”  [Gardner  1985].  Minsky  at  M.I.T.  saw  human  thought  and  intelligence 
as  multifaceted  functions  interacting  with  each  other  to  perform  complex  tasks. 
In  his  book,  Perceptron  (1969),  he  theorized  that  computing  machines  at  the 
time  were  built  upon  erroneous  concepts  and  that,  it  was  necessary  to  provide 
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the  system  with  informative  feedback  about,  successes  and  failures.  This  line 
of  research  led  to  his  seminal  study  of  artificial  neural  networks. 

At  the  same  time  that  AI  research  and  the  study  of  expert  systems  was  ex¬ 
ploring  the  problem-solving  process  from  a  logical  viewpoint,  Torrance  (1915- 
2002)  was  conducting  research  into  the  more  human,  creative  side  of  thinking 
and  critical  thought.  He  defined  creative  thinking  as  “the  process  of  sensing 
difficulties,  problems,  gaps  in  information,  missing  elements,  something  askew; 
making  guesses  and  hypotheses  about  the  solution  of  these  deficiencies;  evalu¬ 
ating  and  testing  these  hypotheses;  possibly  revising  and  restating  them;  and 
finally  communicating  the  result"  [Shaughnessy  1998].  He  believed  that  when 
one  confronts  a  problem  for  which  there  is  no  learned  and  practiced  solution, 
some  creativity  is  necessary.  As  a  result,  in  1974,  he  produced  the  Torrance 
Tests  of  Creative  Thinking  and  suggested  that  intelligence  can  be  measured 
in  ways  other  than  with  an  IQ  test.  The  tests  built  on  the  work  of  Guilford 
and  his  belief  that  the  first  step  in  the  problem-solving  process,  analyzing  the 
problem,  requires  creativity  and  divergent  thinking. 

According  to  Ericsson,  this  creative  aspect  of  expert  performance  is  not 
an  innate  talent  unique  to  select  individuals,  but  one  that  can  be  learned  and 
enhanced  through  “deep  knowledge  of  the  domain  (that)  will  allow  successful 
contributors  to  direct  their  efforts  with  greater  effectiveness”  [Ericsson  1999]. 
He  expanded  on  Watson’s  theory  that  expertise  can  be  obtained  in  any  domain 
after  many  hours  of  practice  and  his  belief  that  “with  a  dozen  healthy  young 
infants  well- formed,  and  my  own  specified  world  to  bring  them  up  in. . . ,  I'll 
guarantee  to  take  any  one  at  random  and  train  him  to  become  any  type  of  spe¬ 
cialist  I  might  select”  [Watson  1919].  To  Ericsson,  expert  performance,  even 
elite  performance,  in  any  domain  is  not  related  to  innate  talent  or  ability  but 
can  be  acquired  by  anyone  through  focused  effort  and  hard  work.  He  believes 
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Figure  2.3:  Model  of  linear  progression  of  expert  performance  [Wood  2000] 


that  in  most  fields,  5  to  10  years  of  experience  is  a  necessary  precondition  for 
becoming  a  master  or  expert;  although,  attainment  is  highly  dependent  on  the 
nature  of  experiences  one  receives  in  that  time  period.  He  and  his  colleagues 
argue  that  the  key  activity  in  the  acquisition  of  expertise  is  deliberate  practice. 
He  says  that  deliberate  practice  requires  “appropriately  challenging  tasks  that 
are  chosen  with  the  goal  of  improving  a  particular  skill.  As  such,  deliberate 
practice  can  be  contrasted  with  activities  such  as  work  and  competitive  per¬ 
formance  in  which  task  demands  and  goals  may  vary  greatly  in  difficulty  and 
fall  beyond  ones  control,  or  play  in  which  the  task  is  relatively  easy  and  is  per¬ 
formed  with  minimal  regard  for  accuracy  or  the  improvement  of  ones  ability” 
[Ericsson  and  Charness  1994]. 

To  Ericsson,  expert  performance  is  not  linear  (see  Figure  2.3);  instead,  the 
10-year  progression  toward  expert  performance  follows  the  path  illustrated  in 
Figure  2.4.  He  and  Chase  argue  that  experts  acquire  long-term  memory  encod¬ 
ing  and  retrieval  skills  and  that  these  skills  expand  the  functional  capacity  of 
working  memory.  Organization  of  knowledge  allows  for  retrieval  rates  typically 
associated  with  short-term  memory.  The  crux  of  his  skilled  memory  theory 
is  that  “skilled  memory  enables  experts  to  rapidly  encode,  store,  and  retrieve 
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Three  Phases  of  Development  oi  Expert  Performance 
Followed  by  o  Qualitatively  Different  Phase  of  Efforts  to 
Attorn  imrnent  Achievements 


Tim*  Sine*  Introduction  to  th*  Domain 


Figure  2.4:  Actual  path  or  phases  of  expert  performance  as  defined  by  Ericsson 
[Ericsson  and  Charness  1994] 


information  within  the  domain  of  their  expertise  and  thereby  to  circumvent 
the  capacity  limitations  that  typically  constrain  novice  performance”  [Ericsson 
and  Staszewski  1989]. 

Unlike  Ericsson,  Sternberg  disagrees  with  Watson’s  deliberate  practice  t  he¬ 
ory.  He  says  that  practice  alone  cannot  account  for  a  Mozart  or  a  Picasso,  that 
practice  is  only  one  part  of  the  picture.  He  believes  instead  that  various  traits 
and  abilities  along  with  practice  lead  to  expertise  [Sternberg  1996].  In  addi¬ 
tion  to  other  traits  and  abilities,  he  ties  the  cognitive  skills  of  problem  solving, 
critical  thinking  and  creativity  into  his  model  of  expertise.  His  model  includes 
the  following  stages:  recognize  or  identify  the  problem;  define  or  represent  the 
problem  mentally  in  different  ways;  develop  a  solution  strategy  (closely  related 
to  divergent  thinking  as  defined  by  Guilford);  organize  knowledge  about  the 
problem;  allocate  mental  or  physical  resources  to  solve  the  problem;  monitor 
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progress  (a  form  of  metacognition),  and  evaluate  the  solution.  The  following 
variables  influence  expert  performance  in  Sternberg’s  model  [Sternberg  2003b]: 

•  Knowledge  or  perspective:  the  ability  to  see  patterns  and  to  frame  prob¬ 
lems 

•  Cognitive  processes  and  strategies:  the  differing  experiences  and  schemas 
a  person  brings  to  the  task 

•  Individual  abilities  and  strategies:  divergent  thinking,  openness,  toler¬ 
ance  of  ambiguity,  and  intrinsic  motivation 

•  External  factors:  the  social  context  in  which  the  problem  is  framed 

Gardner  (1943-  )  investigates  how  human  beings  think.  In  contrast  to 
Spearman's  more  narrow  view,  he  believes  that  intelligence  is  not  a  “single 
entity,”  and  unlike  Binet,  believes  that  an  IQ  test  alone  cannot  measure  the 
multitude  of  factors  involved.  He  also  builds  on  the  cognitive  development  work 
of  Piaget  and  is  not  ready  to  abandon  the  generalist  approach.  Instead,  he 
posits  that  intelligence  is  multifaceted  and  that  independent  traits  of  cognitive 
development  should  be  explored. 

In  the  heyday  of  the  psychometric  and  behaviorist  eras,  it  was 
generally  believed  that  intelligence  was  a  single  entity  that  was 
inherited;  and  that  human  beings. . .  could  be  trained  to  learn  any¬ 
thing _ Nowadays  an  increasing  number  of  researchers  believe  pre¬ 

cisely  the  opposite;  that  there  exists  a  multitude  of  intelligences, 
quite  independent  of  each  other. . .  and  that  it  is  unexpectedly  dif¬ 
ficult  to  teach  things  that  go  against  early  “naive”  theories  that 
challenge  the  natural  lines  of  force  within  an  intelligence  and  its 
matching  domains  [Gardner  1993]. 
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Gardner  believes  that  a  person  has  many  intelligences  that  blend  together 
to  make  each  individual  unique.  In  Frames  of  the  Mind  (1983),  he  formulated 
a  list  of  7  intelligences  [Gardner  1993]: 

•  Linguistic  intelligence  is  the  ability  to  learn  language  and  to  use  it  to 
attain  goals. 

•  Logical- mathematical  intelligence  “entails  the  ability  to  detect  patterns, 
reason  deductively  and  think  logically”  [Gardner  1993].  It  is  usually 
associated  with  scientific  and  mathematical  thinking. 

•  Musical  intelligence  involves  skills  in  all  areas  of  music  including  compo¬ 
sition  and  performance. 

•  Bodily-kinesthetic  intelligence  involves  solving  problems  by  using  the 
mind  or  mental  abilities  in  coordination  with  body  movements. 

•  Spatial  intelligence  is  the  ability  to  recognize  and  use  patterns  found  in 
spaces  of  varying  sizes  and  shapes. 

•  Interpersonal  intelligence  enables  a  person  to  work  well  with  others  and 
to  recognize  and  understand  the  driving  forces  within  them. 

•  Intrapersonal  intelligence  is  the  ability  to  understand  self  to  the  point  of 
being  able  to  appreciate  and  regulate  one’s  life. 

He  says  that  schools  too  often  take  a  single  approach  to  educating  children, 
usually  geared  to  successful  completion  of  uniform  tests  and  typically  measure 
students’  linguistic  and  mathematical  intelligences  at  the  exclusion  of  teaching 
the  high-order  thinking  skills.  He  promotes  the  concept  of  “education  for 
understanding”  in  which  students  are  taught  to  think  critically  and  to  apply 
their  knowledge  to  new  situations.  In  Five  Minds  for  the  Future,  (2007),  he 
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delineates  cognitive  abilities  that  lead  to  desirable  expert  performance.  He 
emphasizes  the  disciplinary  mind,  the  person  who  has  mastery  of  areas  such 
as  science,  mathematics  and  history,  and  of  at  least  one  professional  skill; 
the  synthesizing  mind,  one  who  can  form  ideas  from  various  disciplines  into 
a  logically  consistent  whole  and  then  communicate  the  process  to  others;  and 
the  creating  mind,  one  who  is  able  to  recognize  and  clarify  new  issues  [Gardner 
2007]. 

Chi,  another  current  researcher  in  the  area  of  expertise,  has  studied  the 
domain  knowledge  of  experts  in  a  variety  of  Helds,  including  computer  science. 
She  has  determined  that  experts  not  only  possess  a  large  amount  of  knowl¬ 
edge  in  their  domains,  but  they  are  also  able  to  uniquely  organize,  utilize  and 
structure  the  knowledge  in  a  useful  manner.  One  of  her  important  studies 
deals  with  how  the  experts  apply  their  domain-speciHc  knowledge  to  be  criti¬ 
cal  thinkers  and  problems  solvers  [Chi  and  Bedard  1992].  She  has  found  that 
the  problem-solving  strategies  used  by  experts  vary  greatly  from  those  of  the 
novices.  The  experts  are  more  adept  at  representing  the  problem  with  more 
detail,  exploration  of  given  facts,  possible  limitations  and  other  implications. 
They  use  different  problem-solving  strategies,  tending  to  work  forward  from 
the  given  information,  searching  for  facts  that  can  be  gleaned  from  the  vari¬ 
ables  presented.  Novices,  on  the  other  hand,  tend  to  work  backward  to  the 
given  problem  statement  with  a  goal  in  mind. 

In  further  studies,  she  has  expanded  her  research  and  examined  the  char¬ 
acteristics  of  experts,  including  computer  programmers  [Chi  and  Glaser  2003], 
Among  computer  professionals,  the  experts  have  the  ability  to  notice  useful 
patterns,  and  they  can  “recall  key  programming  language  words  in  meaningful 
clusters”  [Chi  and  Glaser  2003].  They  are  able  to  represent  the  problem  at 
a  detailed  level,  whereas  the  novices  represent  problems  at  a  superficial  level. 
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She  has  further  determined  that  expert  programmers  sort  problems  by  cate¬ 
gorizing  them  by  the  algorithms  needed,  while  novices  sort  them  by  areas  of 
application.  In  agreement  with  her  previous  work,  she  has  also  found  that 
experts  spend  more  time  at  the  beginning  of  the  problem-solving  process  and 
use  greater  critical-thinking  skills  in  trying  to  understand  and  to  analyze  the 
problem;  whereas,  novices  tend  to  spend  little  time  in  the  beginning  phases  of 
the  process  and  prefer  to  generate  a  solution  with  equations  and  algorithms 
immediately. 

Chi’s  studies  have  looked  directly  at  how  an  expert  performs,  but  according 
to  her,  this  type  of  research  may  bring  up  questions  concerning  ways  to  classify 
an  individual  as  an  expert:  “How  does  one  define  an  expert?”,  “What  qualities 
make  a  person  an  expert?”,  “What  distinguishes  that  person  from  the  masses?”, 
“Who  determines  the  ‘expert’  population  versus  the  ‘normal’  population?”.  She 
sees  the  need  for  a  ranking  or  a  measurement  as  a  way  to  classify  the  study 
groups  and  to  answer  some  of  the  questions;  thereby,  she  has  developed  her 
own  classification  system  for  identifying  the  different  levels  of  skills  leading  to 
expertise  [Chi  2006]: 

•  Naive:  one  who  is  totally  ignorant  of  the  domain 

•  Novice:  one  who  is  new  to  the  domain 

•  Initiate:  a  novice  with  initial  training 

•  Apprentice:  a  student  working  with  a  person  skilled  at  the  task 

•  Journeyman:  one  able  to  perform  daily  tasks  unsupervised 

•  Expert:  a  distinguished  or  brilliant  journeyman  having  special  skills  or 
knowledge  derived  from  extensive  experience  with  the  subject  matter 
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•  Master:  an  expert  qualified  to  teach  those  at  a  lower  level  and  regarded 
by  other  experts  as  “the  expert,”  especially  regarding  subject-matter 
knowledge 

Chi  believes  that  it  is  important  for  scientists  and  researchers  studying  ex¬ 
pertise  and  expert  performance  to  be  clear  about  what  they  are  studying  and 
points  out  two  methods  of  research.  The  absolute  method  explicitly  studies 
how  true  experts  perform  using  a  classification  system  such  as  hers.  The  rel¬ 
ative  method,  on  the  other  hand,  studies  the  differences  between  experts  and 
novices.  This  type  of  research,  according  to  Chi,  allows  for  a  much  looser  inter¬ 
pretation  and  definition  of  “expert.”  “Proficiency  level  can  be  grossly  assessed 
by  measures  such  as  academic  qualifications  (undergraduates  versus  graduate 
students),  seniority  or  years  performing  the  task,  or  consensus  among  peers. . . . 
Thus,  the  goal  of  studying  relative  expertise  is  not  merely  to  describe  and  iden¬ 
tify  the  ways  in  which  experts  excel.  Rather,  the  goal  is  to  understand  how 
experts  become  that  way  so  that  others  can  learn  to  become  more  skilled  and 
knowledgeable”  [Chi  2006].  The  study  presented  in  this  dissertation  follows 
the  relative  method  described  above,  exploring  different  groups  of  populations 
delineated  by  characteristics  such  as  position,  rank  and  education  and  how 
they  perform  in  relation  to  each  other. 

2.2  Critical  Thinking  in  Computer  Science 

The  importance  of  teaching  critical-thinking  skills  has  lately  been  re-energized 
and  brought  to  the  forefront  in  educational  studies.  A  consensus  of  higher 
education  associations  in  2004  highlighted  critical  thinking  as  one  of  the  6 
major  intellectual  and  practical  skills  all  college  students  should  develop  [NSB 
2006;  Office  of  Outcomes  Assessment  2006].  Also  in  2004,  the  President  of 
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Harvard  reported  that  over  90  percent  of  the  faculty  consider  critical  thinking 
as  the  most  important  part  of  education.  Nevertheless,  many  faculty  mem¬ 
bers  “spend  most  of  the  time  in  their  curricular  reviews  arguing  over  which 
courses  to  offer  and  which  to  require.  Researchers,  in  contrast,  find  that  the 
arrangement  of  courses  per  se  has  little  effect  on  the  development  of  critical 
thinking”  [Bok  2005].  An  academic  profile  produced  by  the  Education  Test¬ 
ing  Service  in  2003-2004  revealed  that  only  6  percent  of  college  seniors  rated 
“proficient”  in  critical-thinking  skills  while  the  vast  majority.  77  percent,  rated 
“not  proficient”  [AAC  2004],  and  a  2006  “No  Child  Left  Behind"  report  stated 
that  “70  percent  of  employers  said  that  high  school  graduates  were  deficient  in 
critical-thinking. .  .skills”  [Thompson  and  Barnes  2006].  In  spite  of  the  general 
consensus  that  critical-thinking  skills  are  important,  high  school  and  under¬ 
graduate  students  still  lack  mastery  of  those  skills. 

Likewise,  those  in  the  STEM  disciplines  recognize  the  development  of 
critical-thinking  skills  as  one  of  the  most  important  objectives  of  undergradu¬ 
ate  education  [Yuri  tech  2004].  Educational  leaders  and  researchers  in  some  of 
the  STEM  disciplines  have  recognized  that  the  development  of  critical- thinking 
skills  in  students  is  important.  However,  based  on  a  comprehensive  literature 
review,  no  research  specifically  into  the  critical-thinking  skills  of  computer 
professionals  and  experts  and  the  impact  upon  undergraduate  computer  sci¬ 
ence  education  has  been  found,  possibly  due  to  the  absence  of  a  clear  under¬ 
standing  of  exactly  what  critical  thinking  is  as  it  relates  to  computer  science. 
Many  terms  including  “problem-solving  skills,”  “cognitive  abilities,”  “higher- 
order  thinking  skills,”  “creative- thinking  abilities”  and  “critical-thinking  skills” 
have  often  been  used  interchangeably.  Because  this  research  specifically  uses 
the  term  “critical  thinking,”  highlighting  some  of  its  most  prominent  definitions 
is  important.  Critical  thinking  is 
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•  an  “active,  persistent,  and  careful  consideration  of  any  belief  or  supposed 
form  of  knowledge  in  the  light  of  the  grounds  that  support  it  and  the 
further  conclusions  to  which  it  tends”  [Dewey  1910]; 

•  the  mental  processes,  strategies  and  representations  people  use  to  solve 
problems,  make  decisions  and  learn  new  concepts  [Sternberg  1986]; 

•  “the  intellectually  disciplined  process  of  actively  and  skillfully  conceptu¬ 
alizing,  applying,  analyzing,  synthesizing,  and/or  evaluating  information 
gathered  from,  or  generated  by,  observation,  experience,  reflection,  rea¬ 
soning,  or  communication,  as  a  guide  to  belief  and  action”  [Seri veil  and 
Paul  2006]; 

•  an  ability  to  evaluate  information  and  opinions  in  a  systematic,  purpose¬ 
ful,  efficient  manner  [Hill  2007]; 

•  “an  essential  tool  of  inquiry;  purposeful,  self- regulatory  judgment  that 
results  in  interpretation,  analysis,  evaluation,  and  inference,  as  well  as 
explanation  of  the  evidential,  conceptual,  methodological,  criteriological, 
or  contextual  considerations  upon  which  that  judgment  is  based”  [Insight 
Assessment  2007], 

The  CCTST  assessment  tool  and  its  definition  of  critical  thinking,  the  final 
definition  above,  is  the  one  used  in  this  study  (see  Appendix  C  for  details) 

“While  there  is  a  great  deal  of  latitude  in  regard  to  definitions  of  critical 
thinking  and  how  those  definitions  are  applied,  several  commonalities  exist. 
Throughout  the  literature,  critical  thinking  is  defined  as  an  active  process 
which  goes  beyond  basic  acquisition  and  memorization  of  information  in  that 
critical  thinking  requires  the  ability  to  recognize  and  rationally  consider  mul¬ 
tiple  concepts  or  elements”  [Office  of  Outcomes  Assessment  2006].  A  person 
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with  the  ability  to  think  critically  is  “habitually  inquisitive,  well-informed, 
trustful  of  reason,  open-minded,  flexible,  fair-minded  in  evaluation,  honest  in 
facing  personal  biases,  prudent  in  making  judgments,  willing  to  reconsider, 
clear  about  issues,  orderly  in  complex  matters,  diligent  in  seeking  relevant 
information,  reasonable  in  the  selection  of  criteria,  focused  in  inquiry,  and 
persistent  in  seeking  results  which  are  as  precise  as  the  subject  and  the  cir¬ 
cumstances  of  inquiry  permit”  [Facione  1990].  These  skills  are  essential  for 
computer  scientists. 

Three  aspects  of  critical  thinking  that  specifically  apply  to  computational 
science  are  clarity,  the  ability  to  understand  the  facts;  accuracy,  the  ability  to 
understand  the  relationship  between  the  facts  and  reality;  and  relevance,  the 
ability  to  identify  only  relevant  information  and  deductions  [Fagin  et  al.  2006], 
In  other  words,  in  addition  to  having  the  knowledge  and  an  understanding 
of  the  methods  and  techniques  specific  to  the  domain,  a  top- tier  performer 
in  computer  science  must  tie  a  critical  thinker;  therefore,  a  computer  science 
degree  program  needs  to  teach  critical  thinking,  must  require  the  students 
to  understand  their  thoughts,  and  to  be  able  to  express  them  in  a  way  that 
a  computer  can  use  them.  Programming  a  computer  requires  the  following 
steps,  each  one  requiring  critical-thinking  skills  [Pamula  2007]: 

•  “Before  one  can  write  a  computer  program  to  do  something,  one  must 
understand  what  the  program  is  supposed  to  accomplish.  Since  the  in¬ 
tended  objectives  of  a  software  system  are  described  in  English,  signifi¬ 
cant.  critical-thinking  skills  are  required  simply  to  understand  what  is  to 
be  done.  . .  .Software  developers  are  required  to  interview  the  intended 
users  of  the  system  to  try  to  determine  what  really  is  needed.  This  is 
often  an  extraordinarily  difficult,  job,  which  requires  quite  sophisticated 
critical-thinking  skills." 
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•  “Students  must  determine,  in  precise  detail,  how  the  objective  deter¬ 
mined  by  the  previous  step  may  be  accomplished.” 

•  “Students  must  express  the  required  steps  as  a  computer  program.  A 
computer  program  is  a  text  in  an  unforgiving  language,  a  programming 
language.  Programming  languages  are  interpreted  more  formally  and 
literally  than  virtually  any  other  language  in  existence.  Syntax  and  se¬ 
mantics  are  rigidly  defined.  Everything  must  be  correct  for  the  program 
to  operate  properly.” 

Being  successful  in  these  steps  requires  sharp  and  wise  critical-thinking  skills 
that  are  imperative  for  students  of  computer  science  to  have  so  they  will  be 
equipped  to  become  experts,  the  top- tier  performers  in  the  field. 

2.3  Previous  Studies  in  Computer  Science 
Expert  Performance 

For  the  most  part,  previous  studies  on  expert  performance  in  computer  science 
concentrated  on  undergraduate  students  and  on  predicting  their  course  grade. 
Prior  to  1975,  many  of  the  research  projects  on  the  subject  of  computer  science 
and  expert  performance  tended  to  explore  the  demographic  background  and 
the  high  school  achievements  of  the  participants.  These  studies  had  limited 
predictive  power.  Between  1975  and  1981  attention  focused  on  specific  pro¬ 
gramming  aptitude  tests  (PATs)  such  as  IBM’s  PAT.  From  1981  to  1990.  most 
studies  explored  various  learning  styles  necessary  for  the  expert  performance 
in  computer  science  [Evans  and  Simkin  1989]. 

Wood  (2000)  notes  that  one  of  the  major  limitations  on  previous  studies 
of  expertise  in  computer  science  was  in  obtaining  large  enough  sample  sizes 
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Figure  2.5:  Review  of  background  literature  and  previous  studies  on  the  factors 
in  expert  performance 


for  data  analysis.  She  examined  36  studies  from  1976  to  1999  and  found  that 
the  “expert”  group  averaged  only  10  participants.  Only  8  had  more  than  20 
participants  in  the  control  group,  and  the  novice  group  averaged  only  13.  She 
identified  the  following  variables:  dependent  var i ables-  memory  (6  studies), 
knowledge  structures  {10  studies),  comprehension  {6  studies),  and  problem 
solving  (6  studies).  Expertise  was  the  sole  independent  variable  in  a  majority 
of  the  studies  [Wood  2000]. 

In  an  attempt  to  find  a  research  project  that  dealt  specifically  with  the 
critical-thinking  skills  of  expert  performers  in  top-tier  computer  science  pro¬ 
fessionals  and  that  compared  their  assessment  scores  with  the  same  scores  of 
undergraduate  students  (see  Figure  2.5),  the  researcher  conducted  a  compre¬ 
hensive  literature  review  (see  the  projects  listed  in  Appendix  D). 
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This  review  of  the  research  suggests  that  past  studies  in  expert  performance 
in  computer  science  focused  primarily  on  the  technical  skills  of  programmers 
and  on  predicting  academic  success  and  achievement  at  the  undergraduate 
level.  The  researcher  found  no  studies  that  specifically  assessed  the  critical- 
thinking  skills  of  top-tier  performers  in  computer  science  and  compared  their 
scores  with  those  of  undergraduate  students.  Considering  the  growing  need 
for  computer  science  graduates  with  cognitive-thinking  skills  compounded  by 
the  decline  in  enrollment  in  computer  science  programs  across  the  nation, 
the  specific  critical-thinking  skills  characteristic  of  expert  computer  science 
professionals  merit  study.  One  of  the  first  steps  in  addressing  this  need  is 
to  compare  and  contrast  the  critical-thinking  skills  of  top-tier  professionals 
and  undergraduate  students,  and  to  assess  correlations  between  currently  used 
measures  of  academic  success  and  these  skills. 
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Chapter  3 
Methodology 


The  study’s  hypothesis  is  that  t,o|>tier  computer  science  professionals  have 
demonstrable,  critical-thinking  abilities  that  are  superior  to  those  of  under¬ 
graduate  students.  The  importance  of  this  primary  hypothesis  is  that  critical- 
thinking  abilities  are  a  learned  behavior  and  not  “naturally”  present.  In  spite 
of  the  large  amount  of  literature  related  to  this  subject,  this  relationship  has 
yet  to  be  validated.  In  order  to  test  the  veracity  of  this  hypothesis,  the  re¬ 
search  presented  here  tested  a  random  sample  of  students,  both  freshmen  and 
seniors,  and  compared  the  assessment  scores  of  these  two  samples  with  each 
other  and  with  those  of  the  experts.  The  hypothesis  is  considered  confirmed 
if  the  professionals  are  statistically  better  than  the  students. 

The  study  reported  here  addresses  the  following  related  research  questions 
that  represent  critical  elements  of  the  primary  hypothesis: 

1.  What  critical-thinking  abilities  do  top-tier  professionals  exhibit? 

2.  What  are  the  critical-thinking  abilities  of  freshmen  students? 

3.  What  are  the  critical-thinking  abilities  of  senior  computer  science  stu¬ 
dents? 

4.  How  do  the  critical-thinking  skills  of  the  freshmen  and  senior  computer 
science  students  compare? 

5.  How  do  currently  used  measures  of  academic  performance,  such  as  grade 
point  average,  correlate  to  this  assessment  of  critical-thinking  skills? 


6.  What  are  the  differences  in  critical-thinking  skills  between  computer  sci¬ 
ence  professionals  and  undergraduate  computer  science  students? 

Figure  3.1  shows  the  process  used  to  answer  these  questions  and  to  meet 
the  research  objectives.  To  achieve  the  overall  objective,  a  series  of  “sul> 
studies"  produced  data  to  evaluate  the  main  hypothesis.  Study  1  addresses 
the  critical-thinking  abilities  of  top-tier  performers,  the  experts  as  defined  by 
their  position,  responsibility,  recognition,  or  rank,  and  is  used  to  establish  a 
benchmark  of  these  abilities  of  top  performers  in  computer  science.  Study 
2  is  the  assessment  of  the  critical-thinking  abilities  of  freshmen  and  senior 
college  students  at  Clemson  University.  To  see  if  these  abilities  are  currently 
being  measured,  study  3  tests  the  data  for  a  correlation  between  these  abilities 
and  grade-point  average,  SAT  scores,  and  credit  hours  earned.  Study  4  is 
the  comparison  of  the  critical- thinking  scores  of  the  students  to  those  of  the 
expert  groups  assessed.  Each  arrow  is  substantiated  through  either  background 
literature,  observations,  or  testing. 

The  remainder  of  this  chapter  details  this  plan  including  a  description  of 
its  implementation,  the  assessment  tool  used  to  answer  the  research  questions, 
and  the  participants  assessed.  It  also  explains  the  data  collection  procedures, 
including  the  relevant,  variables,  and  the  statistical  tests  used  to  analyze  the 
data. 


3.1  Population  and  Samples 

This  research  project  tested  6  different  population  samples,  college  freshmen 
and  seniors,  faculty  members  in  the  STEM  disciplines,  computer  profession¬ 
als  in  industry  and  in  the  military,  and  a  group  of  military  officers  with 
non-technical  backgrounds  representing  non-industry  professionals  outside  the 
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Figure  3.1:  Research  plan  of  attack 


*12 


STEM  disciplines.  Participation  in  the  study  was  voluntary,  and  a  Clemson 
Institution  Review  Board  (IRB)  (#  06-IRB-226)  reviewed  and  approved  it 
before  it  commenced  {see  Appendix  H). 

All  undergraduate  participants,  both  the  freshmen  and  seniors,  were  full¬ 
time  students  at  Clemson  University  during  the  Spring  of  2007.  Class  presenta¬ 
tions  to  5  first-year  computer  science  classes  and  4  first-year  ROTC  leadership 
classes  resulted  in  the  51  freshman  volunteers.  This  sample  represented  a  wide 
range  of  students  in  their  first  year  of  college  intending  to  major  in  various 
areas,  primarily  in  STEM  disciplines.  Specifically,  the  sample  included  15  ma¬ 
joring  in  computer  science,  3  in  mathematics,  2  in  physics,  2  in  chemistry,  2 
in  civil  engineering,  3  in  biology,  4  in  the  behavioral  sciences,  2  in  mechanical 
engineering,  10  in  general  engineering,  2  in  business,  3  in  foreign  studies,  1  in 
history,  and  2  in  political  science.  The  14  female  and  41  male  participants  in¬ 
cluded  4  Asians,  2  African-Americans,  40  Caucasians,  and  6  who  elected  not  to 
identify  their  ethnic  backgrounds.  The  mean  age  of  these  freshman  volunteers 
was  19.43  years,  with  the  youngest  being  17  and  the  oldest  43. 

In-class  presentations  to  computer  science  majors  and  an  email  announce¬ 
ment  sent  to  senior  math  and  engineering  majors  obtained  the  senior  partici¬ 
pants.  The  majority  of  the  resulting  sample  were  computer  science  majors  en¬ 
rolled  in  senior- level  computer  science  courses.  They  consisted  of  27  computer 
science  majors,  2  mathematical  science  majors,  and  1  electrical  engineering 
major.  The  6  female  and  24  male  participants  included  1  Asian,  3  African- 
Americans  and  26  Caucasians.  The  mean  age  of  these  senior  participants  was 
21.93  years,  with  the  youngest  being  20  and  the  oldest  26. 

The  faculty  participants  were  professors  in  computer  science  and  related 
STEM  disciplines  from  several  major  universities  solicited  via  an  email  an¬ 
nouncement.  The  expertise  level  of  faculty  members  was  difficult  to  gauge; 
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therefore,  all  were  automatically  considered  expert  participants,  regardless  of 
school  or  academic  rank.  The  participants  consisted  of  9  computer  science 
faculty,  3  faculty  members  from  mathematical  science,  3  of  physics,  2  of  com¬ 
puter  engineering,  1  of  systems  engineering,  1  of  industrial  engineering  and  1 
of  chemistry.  The  6  female  and  14  male  faculty  included  2  African-Americans, 
2  Caucasians,  and  2  who  chose  not  to  reveal  their  ethnic  backgrounds.  The 
mean  age  of  the  faculty  members  was  45.3  years,  with  the  youngest  being  22 
and  the  oldest  67. 

To  obtain  industry  participants,  a  facilitator  in  each  of  6  different  computer- 
related  companies  was  contacted  and  sent  an  informational  packet  and  on-line 
testing  procedures  as  well  as  a  set  of  user  names  and  passwords  to  distribute 
to  the  volunteers  within  their  organizations.  This  group  consisted  of  1  female 
and  18  male  volunteers  including  1  Hispanic,  2  Asians,  1  African-American, 
12  Caucasians  and  2  who  chose  not  to  identify  their  ethnic  backgrounds.  'Flic 
mean  age  of  the  participants  was  33.06  years  with  the  youngest  being  24  and 
the  oldest  55. 

The  United  States  Air  Force  participants  were  from  computer  and  commu¬ 
nication  disciplines  and  from  the  Defense  Information  Systems  Agency  (DISA) 
(see  Appendix  F.l).  Subjects  primarily  consisted  of  field  grade  officers  in  the 
communications  field,  and  included  signal  officers,  automators,  engineers,  and 
communications-coinputer  officers.  The  United  States  Air  Force  communica¬ 
tions  officers  were  chosen  as  expert  participants  in  this  study  because  they 
demonstrate  high  levels  of  achievement  and  are  representatives  of  an  elite  per¬ 
forming  group.  They  are  all  specialists  in  their  field  and  were  hand-selected  for 
their  assignments.  They  are  mainly  field-grade  officers  with  10  or  more  years 
of  experience  in  high-tech  leadership  positions.  This  group  included  4  females 
and  23  males  including  2  African-Americans,  23  Caucasians,  and  2  who  chose 
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Table  3.1:  Mean  age  of  sample  groups  in  current  study 


Group 

n 

Mean  Age 

StDev 

Min 

Max 

Freshmen 

54 

19.43 

4.12 

17 

43 

Senior 

29 

21.93 

1.93 

20 

26 

Faculty 

20 

45.35 

14.29 

22 

67 

Industry 

18 

33.06 

9.41 

24 

55 

Air  Force 

27 

38.07 

6.59 

27 

50 

Army 

31 

33.00 

6.23 

23 

50 

not  to  reveal  their  ethnic  backgrounds.  The  mean  age  of  the  participants  from 
the  Air  Force  was  38.07  years  with  the  youngest  being  27  and  oldest  50. 

The  final  group  of  participants  was  from  the  United  States  Signal  Corps 
Army  Officers  located  at  Fort  Gordon,  Georgia  (see  Appendix  F.2).  This  group 
represents  officers  in  a  first-  level  information  technology  school,  all  have  4-year 
college  degrees  and  are  experienced  Army  officers.  It  consisted  of  2  females  and 
20  males  including  7  African-Americans,  4  Asians,  5  Hispanic-Americans,  11 
Caucasians,  and  3  who  chose  not  to  reveal  their  ethnic  bac  kgrounds.  The  mean 
age  of  the  participants  from  Fort  Gordon  was  33.00  years,  with  the  youngest 
being  23  and  the  oldest  50. 


3.2  Measurements  and  Instrumentation 

A  study  of  creativity  skills  was  initially  considered  for  this  research;  however, 
most  tests  had  objective  grading  criteria  and  many  required  the  grader  to  at¬ 
tend  training  in  order  to  interpret  the  results.  None  had  creativity  questions 
relating  specifically  to  computer  science,  and  generating  them  would  have  re¬ 
quired  their  being  somehow  validated.  Some  of  these  tests  had  varying  options, 
but  with  limitations  (see  Appendix  A). 
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Instead  of  creativity,  critical  thinking  was  chosen  as  the  focus  of  this  study 
and  the  California  Critical  Thinking  Skills  Test  {CCTST)  version  2000,  de¬ 
veloped  by  Peter  Fraeione,  was  selected  as  the  assessment  tool  [Facione  1090; 
Insight  Assessment  2007], In  addition  to  the  CCTST,  several  other  critical- 
thinking  assessments  were  considered  for  this  research.  The  following  criteria 
were  established  in  selecting  an  appropriate  assessment: 

•  A  nationally  validated  test 

•  Capable  of  measuring  subcategories  of  critical  thinking 

•  Requiring  no  special  training  to  grade  and  interpret  results 

•  Administered  on-line 

•  Can  be  completed  in  approximately  45-minutes  or  less 

The  CCTST,  a  34-question,  multiple-choice,  on-line  assessment,  allows  for 
immediate  feedback  to  the  test-taker  on  5  areas  of  critical  thinking:  induc¬ 
tive  reasoning,  deductive  reasoning,  analysis,  inference,  evaluation,  and  total 
critical-thinking  score.  It  is  time  efficient,  convenient,  and  can  be  easily  ac¬ 
cessed  by  all  participants  as  well  as  by  the  researcher,  and,  according  to  Insight 
Assessment,  the  KR-20  alphas  range  from  0.78  to  0.84,  indicating  a  high  level 
of  internal  consistency.  Appendix  C  provides  details  about  the  test,  including 
sample  questions. 

3.3  Data  Collection 

The  researcher  used  on-line  testing  procedures  to  assess  the  student  partici¬ 
pants.  He  emailed  each  student  a  cover  letter  and  an  instruction  sheet,  in¬ 
cluding  a  login  name  and  password  pair  (see  Appendix  G).  Testing  began  on 
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January  12,  2007,  and  all  tests  were  completed  on  or  before  May  1,  2007.  The 
assessment  was  untimed,  and  volunteers  took  the  test  when  and  where  they 
chose. 


3.4  Variables 

In  comparing  the  difference  of  means  between  samples  (research  questions  1-4 
and  6)  the  sole  independent  variable:  study  population  represents  class  year 
for  student  participants  and  professional  group  for  military  and  industry  par¬ 
ticipants.  The  independent  variables  used  in  this  study  for  research  question 
5  and  listed  in  Table  3.2  are 

•  Math  SAT  is  the  participant’s  score  on  the  mathematical  section  of  the 
Scholastic  Aptitude  Test.  This  score  was  not  always  available. 

•  Verbal  SAT  is  the  participant’s  score  on  the  verbal  section  of  the  Scholas¬ 
tic  Aptitude  Test.  This  score  was  not  always  available. 

•  Current  GPA  is  the  current  grade  point  average  for  the  student  partici¬ 
pants. 

•  Gender  is  voluntary  information  requested  on  the  student  survey. 

•  Age 

•  Race  is  the  ethnicity  of  the  participant,  obtained  from  voluntary  infor¬ 
mation  on  the  participant  survey. 

•  University 

•  Major  is  the  major  or  intended  major  of  the  undergraduate  participant. 
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•  Study  Population  is  the  participant’s  population:  freshman,  senior,  fac¬ 


ulty,  industry,  air  force,  or  army. 


•  Group  is  a  subjective  grouping  of  the  participants  based  on  the  field  of 
work  for  industry  and  military  professionals  or  academic  major  for  the 
student  participants. 


Table  3.2:  Independent  variables  for  research  question  5 


Name 

Descript  ion 

Source 

Math  SAT 

Verbal  SAT 

Current  GPA 

Gender 

Ago 

Race 

University 

Major/ intended  Major 
Study  Population 

Group 

Continuous  -  Scholastic  Aptitude  Teat  Math  Score 
Continuous  -  Scholastic  Aptitude  Test  Verbal  Score 
Continuous  -  Current  GPA  of  student  participants 
Dichotomous  -  Gender  of  participant 

Discrete  -  Age  of  participant 

Nominal  -  Ethnicity  of  participant 

Nominal  -  Undergraduate  college  of  participant 
Nominal  -  Undergraduate  major 

Discrete  -  Population  sample  of  participant 

Nominal  -  Main  area  of  work  or  major  of  participant 

Registrar 

Registrar 

Registrar 

Participant  survey 
Participant  survey 
Participant  survey 
Participant  survey 
Participant  survey 
Assigned 

Assigned 

The  dependent  variables  used  in  this  study  and  listed  in  Table  3.3  are 

•  CCTST  Induction  Score  represents  the  participant's  score  on  the  induc¬ 
tive  reasoning  portion  of  the  CCTST,  which  has  a  maximum  score  of 
17. 

•  CCTST  Deduction  Score  represents  the  participant’s  score  on  the  de¬ 
ductive  reasoning  portion  of  the  CCTST,  which  has  a  maximum  score  of 
17. 

•  CCTST  Analysis  Score  represents  the  score  on  the  analysis  portion  of 
the  CCTST,  which  has  a  maximum  score  of  7. 

•  CCTST  Inference  Score  represents  the  participant’s  score  on  the  infer¬ 
ence  portion  of  the  CCTST,  which  has  a  maximum  score  of  16. 

•  CCTST  Evaluation  Score  represents  the  participant’s  score  on  the  eval¬ 
uation  portion  of  the  CCTST,  which  has  a  maximum  score  of  11. 
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•  CCTST  Total  Score  represents  the  participant’s  total  critical- thinking 
score  on  the  CCTST,  which  has  a  maximum  score  of  34. 


Table  3.3:  Dependent  Variables  for  all  research  questions 


Variable  Name 

Description 

Source 

CCTST  Induction  Score 
CCTST  Deduction  Score 
CCTST  Analysis  Score 
CCTST  Inference  Score 
CCTST  Evaluation  Score 
CCTST  Total  Score 

Discrete  variable  representing  CCTST  score 
Discrete  variable  representing  CCTST  score 
Discrete  variable  representing  CCTST  score 
Discrete  variable  representing  CCTST  score 
Discrete  variable  representing  CCTST  score 
Discrete  variable  representing  CCTST  score 

CCTST 

CCTST 

CCTST 

CCTST 

CCTST 

CCTST 

3.5  Data  Analysis 

The  statistical  software  package  Minitab  version  15.1  1  [Minitab  Inc.  2007], 
performed  data  analysis  and  produced  textual  and  graphical  models.  The  re¬ 
searcher  used  the  following  statistical  tests,  all  using  a  0.05  level  of  significance, 
to  analyze  the  data  [Brown  2007]: 

1.  Computing  means  and  standard  deviations  for  each  area  measured  by 
the  CCTST  assessment  generated  a  benchmark  for  the  critical- thinking 
skills  of  professionals  (Research  Question  1). 

2.  Means  and  standard  deviations  determined  critical-thinking  abilities  of 
freshmen  and  seniors  (Research  Questions  2  and  3). 

3.  A  one-way  analysis  of  variance  (A NOVA)  determined  if  significant  dif¬ 
ferences  in  critical- thinking  skills  exist  between  freshmen  and  seniors, 
between  freshmen  and  experts,  and  between  seniors  and  experts.  Addi¬ 
tionally,  Hsu’s  Comparison  test  identified  factor  levels  that  are  the  best, 

'Available  at  www.minitab.com 
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are  insignificantly  different  from  the  best,  and  are  significantly  different 
from  the  best  [Minitab  Inc.  2007].  Hsu’s  comparison  creates  a  confi¬ 
dence  interval  for  the  difference  between  each  level  mean  and  the  best 
of  the  remaining  level  means.  An  interval  that  has  zero  as  an  end  point 
indicates  a  statistically  significant  difference  between  the  corresponding 
means.  Specifically  [Hsu  1996]: 

•  Confidence  interval  contains  zero  -  No  difference 

•  Confidence  interval  entirely  above  zero  -  Significantly  better 

•  Confidence  interval  entirely  below  zero  -  Significantly  worse 

(Research  Questions  4  and  6). 

4.  Computation  of  a  Pearson  Product-Moment,  correlation  tested  the  null 
hypothesis  that  no  significant  correlation  exists  between  student  GPA. 
SAT  and  critical-thinking  skills  (Research  Question  5). 
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Chapter  4 

Experimental  Results 


This  study  used  the  assessment  tool  CCTST  by  Insight  Assessment  t.o  eval¬ 
uate  the  hypothesis  that  computer  science  experts  have  exceptional  critical- 
thinking  skills  greater  than  those  of  undergraduates.  It  focuses  on  addressing 
the  data  relevant  to  the  research  questions  posed  in  Chapter  3  concerning 
the  assessment  and  comparison  of  critical-th  inking  skills  of  top-tier  computer 
science  performers  and  college  freshmen  and  seniors.  Section  4.1  reports  the 
response  rates  of  the  various  populations  to  the  requests  for  participation  in 
the  survey.  Section  4.2  addresses  the  survey  findings  as  they  relate  to  the  re¬ 
search  questions  and  concludes  with  additional  information  supported  by  the 
data  collected  from  the  surveys. 


4.1  Response  rate 

In  order  to  obtain  a  sufficient  sample  size  for  each  of  the  populations,  a  large 
number  of  students,  faculty  members,  industry  experts  and  military  personnel 
received  inquiries  about  participating  in  the  survey.  As  reported  in  previous 
study’s  [Wood  2000],  finding  volunteer  participants  is  one  of  the  greatest  obsta¬ 
cles  encountered  when  conducting  this  type  of  research.  The  overall  response 
rate  to  this  study’s  request  for  participation  was  47.29%  (166  out  of  351}. 

The  response  rate  of  undergraduate  students  to  the  initial  request  was 
3.03%  (5  out  of  165)  with  no  incentives  offered.  With  an  incentive  of  drink 
and  food  coupons  or  small  amounts  of  extra  credit,  the  response  rate  improved 
significantly.  Of  the  120  freshmen  contacted,  55  participated  for  a  45.83% 


response  rate,  and  30  of  the  45  seniors  contacted  participated  for  a  66.67% 
rate. 

The  response  rate  for  the  faculty  was  much  lower.  In  answer  to  the  36 
email  announcements  sent,  only  4  individuals  responded  and  participated  in 
the  survey  for  a  response  rate  of  11.11%.  The  additional  faculty  volunteers 
came  from  an  email  solicitation  sent  to  the  contacts  on  a  STEM-interest  email 
list. 

The  response  rates  for  industry  participants  was  also  low.  Out  of  several 
companies  contacted,  only  6  responded  and  requested  assessment  surveys.  Of 
the  60  surveys  distributed,  18  returned  for  a  response  rate  of  30%.  Two  com¬ 
panies  required  a  non-disclosure  statement,  and  all  chose  not  to  be  individually 
identified  in  the  study. 

Military  personnel  and  leadership  were,  for  the  most  part,  eager  to  help 
with  the  study  and  to  participate.  The  United  States  military  is  actively  in¬ 
terested  in  critical-thinking  skills,  and  this  research  particularly  interested  the 
leadership  and  personnel  at  the  agencies  contacted.  Of  the  45  surveys  dis¬ 
tributed  to  the  United  States  Air  Force  Communications  Officers,  28  returned 
for  a  response  rate  of  62.22%.  The  response  rate  from  the  United  States  Army 
Signal  Corps  was  similar  to  that  of  the  Air  Force  with  45  surveys  distributed 
and  31  returned  for  a  response  rate  of  68.89%. 


4.2  Findings 

4.2.1  Experts’  Critical-Thinking  Scores 

The  following  data  is  in  response  to  Research  Question  1  concerning  the 
critical-thinking  abilities,  as  measured  by  the  CCTST,  of  top-tier  performers, 
the  experts. 
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Table  4,1:  Mean  total  scores  for  the  expert  populations 


GROUP 

11 

Mean 

StDev 

Min 

Med 

Max 

Faculty 

20 

26.100 

4.191 

17 

26 

32 

Industry 

18 

25.278 

3.268 

20 

26 

31 

Air  Force 

27 

26.556 

4.003 

12 

27 

31 

Air  Force0 

25 

27.480 

2.143 

24 

28 

31 

“Air  Force  scores  with  outliers  removed 


The  average  mean  score  on  the  CCTST  of  each  of  the  3  expert  populations, 
is  significantly  above  the  average  mean  score  of  16.801  reported  by  Insight 
Assessment  (N=2061,  SD=5.062)  (see  Appendix  J).  No  statistical  difference 
showed  between  the  3  groups  at  the  0.05  level.  Table  4.1  shows  the  mean  total 
critical-thinking  score  for  each  of  the  3  expert  populations  as  measured  by  the 
assessment. 

Figure  4.1  shows  a  normal  distribution  (A-Squared  =  0.46,  P- Value  = 
0.231)  with  a  median  of  26  (M— 26.10)  for  the  total  critical-thinking  scores  for 
the  faculty  population. 

Figure  4.2  shows  the  results  of  the  critical-thinking  scores  for  the  industry 
participants.  Their  scores  also  show  a  normal  distribution  (A-Squared  =  0.43, 
P- Value  =  0.273),  with  a  median  of  26  (M=25.28). 

Figure  4.3  shows  the  results  of  the  critical-thinking  scores  for  the  Air  Force 
expert  participants,  including  2  outlying  scores  as  calculated  by  Minitab.  The 
median  score  for  these  participants  is  27  (M=26.56).  Figure  4.4  shows  t  he 
results  of  the  scores  for  the  Air  Force  expert  participants  with  the  outlying 
scores  removed.  The  Anderson- Darling  Normality  Test  shows  that  this  is  a 
normal  distribution  (A-Squared  —  0.52,  P- Value  —  0. 169)  with  a  median  score 
of  28  (M=27.48).  The  Air  Force  experts'  raw  score  was  higher  than  the  scores 
of  the  other  experts;  however,  there  is  no  measurable  statistical  difference. 
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Figure  4.1:  Mean  total  score  for  the  faculty  population 
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Figure  4.2:  Mean  total  score  for  the  industry  participants 
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Figure  4.3:  Mean  total  score  for  Air  Force  participants  with  outlying  scores 
indicated  by  *  at  the  12  and  18  levels 


Summary  for  TOTAL 

GROUP  =  AF  Comm 


Anders  cxvDariing  Normality  last 

Squared 

0.52 

P-Value 

0.169 

Mean 

27.480 

StDev 

2.143 

Variance 

4.593 

Skewness 

■0.16337 

Kurt  oils 

-1,11547 

N 

25 

Minimum 

244300 

lit  Quartik 

26  4300 

Median 

284300 

3rd  Quart ile 

29,500 

Maximum 

314300 

95%  Confidence  Interval  for  Mean 

26.595 

28.365 

95%  Confidence  Interval  for  Median 

26.000 

29  4)00 

95%  Confidence  Interval  for  StDev 

1.673 

2582 

ua  270  273  3L0  313  2M 


Figure  4.4:  Mean  total  score  for  the  Air  Force  participants  with  outlying  scores 
removed 
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Table  4.2:  Mean  total  scores  in  the  sub-categories  for  the  expert  populations 


GROUP 

Ind(17) 

Ded(17) 

Anl(7) 

Inf(16) 

Eval(ll) 

Tot(34) 

Faculty 

13.90 

12.20 

6.35 

12.10 

7.65 

26.10 

Industry 

12.61 

12.67 

5.67 

12.61 

7.00 

25.28 

Air  Force 

13.59 

12.89 

5.93 

12.63 

8.11 

26.56 

Air  Force0 

14.04 

13.36 

6.04 

13.00 

8.56 

27.48 

°AF  expert  population  with  outliers  removed 

The  number  in  the  parenthesis  indicates  the  maximum  possible  score 


Table  4.2  shows  the  results  for  each  of  the  expert  populations  in  the  critical- 
thinking  sub-categories:  induction,  deduction,  analysis,  inference,  evaluation, 
and  the  total  as  reported  by  Insight  Assessment.  There  is  no  statistical  dif¬ 
ference  in  any  of  the  categories  between  the  scores  of  the  expert  populations 
sampled. 

4.2.2  Students’  Critical-Thinking  Scores 

The  following  data  is  in  response  to  Research  Questions  2,  3,  and  4  concerning 
the  critical-thinking  skills  of  students. 

Table  4.3  shows  the  data  collected  in  the  sub-categories  of  the  critical- 
thinking  abilities  of  freshmen.  The  mean  score  of  the  freshmen  participants, 
20.981  (SD=5.023,  N=54),  was  significantly  higher  than  the  average  score  re¬ 
ported  by  Insight  Assessment  (M=16.801,  N=2061,  SD=5.062){see  Figure  4.5). 

The  data  collected  in  response  to  Research  Question  3  concerning  the 
critical-thinking  abilities  of  seniors,  shows  that  the  mean  score  of  the  senior 
participants,  24.931  (SD=4.869.  N=29),  was  also  significantly  higher  than  the 
average  score  reported  by  Insight.  Assessment  (M=  16.801,  N=2061,  SD =5.002} 
{see  Figure  4.6).  Table  4.4  shows  the  total  scores  in  the  critical-thinking  sub¬ 
categories  for  the  senior  participants  as  reported  by  Insight  Assessment. 
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Summary  for  TOTAL 

GROUP  =  Fresh 


AndenonDariing  Test 

A*  Squared 

0.27 

P-V*k» 

0*79 

Mean 

20,961 

StD*v 

5023 

Variance 

25.226 

Skewness 

0,367243 

Kurtoih 

0.158551 

N 

54 

Minimum 

6X100 

Is*  Quartile 

17750 

Median 

21000 

3rd  Quirt  lift 

24750 

MilUTlUITl 

50.000 

95%  Confidence  Interval  for  Mam 

19011 

22  352 

95%  Confidence  Interval  for  Median 
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23000 

95%  Confidence  Interval  for  StDev 

4,222 
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22 


n 


Figure  4.5:  Mean  total  scores  for  the  freshmen  participants 


Summary  for  TOTAL 

GROUP  =  Senior 


Anderson- Darling  Ncrmalitv  Test 

A-Squared 

0.68 

P  Value 

0068 

Mean 

24,931 

StDev 

4,869 

Variance 

23709 

Skewness 

-0,852163 

Kuttosis 

0.550692 

N 

29 

Minimum 

13,000 

1st  Quartile 

22,000 

Median 

26  000 

3rd  Quartile 

28,000 

Meiimum 

33  000 

95%  Confidence  Interval  for  Mean 

23,079 

26,783 

95%  Confidence  Interval  for  Median 

24,000 

28,000 

95%  Confidence  Interval  for  SiDev 

3064 

6.505 

Figure  4.6:  Mean  total  score  for  the  senior  participants 
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Table  4.3:  Mean  total  scores  in  the  sub-categories  for  the  freshmen  participants 


Variable 

n 

Mean 

StDev 

Var 

Min 

Med 

Max 

Induction 

54 

11.074 

2.433 

5.919 

4 

1 1 

13 

Deduction 

54 

9.907 

3.217 

10.35 

2 

10 

12 

Analysis 

54 

5.019 

1.407 

1.981 

1 

5 

6 

Inference 

54 

10.074 

2.906 

8.447 

3 

10 

11 

Evaluation 

54 

5.889 

1.959 

3.836 

2 

6 

8 

TOTAL 

54 

20.981 

5.023 

25.226 

6 

21 

26 

Table  4.4:  Mean  total  scores  in  the  sub-categories  for  the  senior  participants 


Variable 

n 

Mean 

StDev 

Var 

Min 

Med 

Max 

Induction 

29 

13.138 

2.279 

5.195 

6 

13 

16 

Deduction 

29 

11.793 

3.353 

11.241 

5 

12 

17 

Analysis 

29 

5.517 

1.379 

1.901 

1 

6 

7 

Inference 

29 

11.690 

2.620 

6.865 

6 

11 

16 

Evaluation 

29 

7.724 

2.016 

4.064 

1 

8 

11 

TOTAL 

29 

24.931 

4.869 

23.709 

13 

26 

33 

Table  4.5  shows  the  data  collected  in  response  to  Research  Question  4 
concerning  the  comparison  of  critical-thinking  skills  of  the  freshmen  and  senior 
participants.  The  two-tailed  T-Test  shows  that  the  seniors  had  statistically 
higher  scores  than  the  freshmen  (P-Value=0.001.  DF=58).  Figure  4.7  shows 
the  mean  total  scores  for  freshmen  and  seniors. 

In  response  to  Research  Question  5,  Pearson’s  Correlation  tests  determined 
the  relationship  between  currently  used  measures  of  academic  performance  and 
the  CCTST.  A  correlation  of  0.696  with  a  P-value  of  0.000  indicates  a  positive 


Table  4.5:  Two-tailed  T  test  of  freshmen  vs.  seniors 


GROUP 

N 

Mean 

StDev 

SE  Mean 

Freshmen 

54 

20.98 

5.02 

0.68 

Seniors 

29 

24.93 

4.87 

0.90 

Difference  —  fi  (Fresh)  -  fi  (Senior) 

Estimate  for  difference:  -3.95 
95%  Cl  for  difference:  (-6.22,  -1.68) 

T-Test  of  diff=0:  T-Val=-3.48  P-Val=0.001  DF=58 
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Figure  4.7:  Mean  total  scores  for  freshmen  and  senior  participants;  the  boxes 
represent  the  first  quarter  through  the  third  quarter  scores,  with  a  line  depict¬ 
ing  the  mean  score 


correlation  between  total  and  verbal  SAT  scores  for  the  student  participants 
(see  Figure  4.8).  Likewise  a  correlation  of  0.693  with  a  P- value  of  0.000  shows 
a  positive  relationship  between  total  and  math  SAT  scores  (see  Figure  4.9). 

Figure  4.10  shows  the  correlation  between  total  scores  and  grade  point 
average,  termed  grade  point  record  (GPR)  at  Clemson  University  (GPR  and 
GPA  are  used  interchangeably  in  this  study).  The  Pearson’s  Correlation  of 
0.301  with  a  P-value  of  0.062  indicates  insufficient  statistical  evidence  to  show 
any  correlation  between  total  critical-thinking  scores  and  GPA. 

Figure  4.11  shows  the  correlation  between  total  score  and  cumulative  hours 
earned.  The  Pearson’s  Correlation  of  0.297  and  a  P-value  of  0.066  shows 
insufficient  statistical  evidence  of  any  correlation  between  the  total  critical- 
thinking  scores  and  cumulative  hours  earned. 
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Scatterplot  of  TOTAL  vs  VERBAL_SAT 


Pearson  correlation  =  0,696,  P-Vakje  =  0.000 
n  =  36 


Figure  4.8:  Correlation  between  total  and  verbal  SAT  scores 


Scatterplot  of  TOTAL  vs  MATH.SAT 


Pearson  correlation  =  0.693,  P- Value  =  0.000 
n  =  36 


Figure  4.9:  Correlation  between  total  and  math  SAT  scores 
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Scatterplot  of  TOTAL  vs  CUM_GPR 


Pearson  correlation  a  0.301,  P- Value  =  0,062 
n  =  39 


Figure  4.10:  Correlation  between  total  scores  and  cumulative  grade-point  av¬ 
erage 


Scatterplot  of  TOTAL  vs  CUMJHRS_EARNED 


Pearson  correlation  =  0.297,  P- Value  =  0,066 
n  -  39 


Figure  4.11:  Correlation  between  total  scores  and  cumulative  credit  hours 
earned 
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Table  4,6:  Pearson’s  Correlation  data  as  published  by  Insight  Assessment,  and 
as  found  in  this  study 


Stat 

IA 

Study 

SAT  Verbal 
SAT  Math 
Age 

College  GPA 

0.55  -  0.62 
0.44  -  0.48 
0.006 
0.20  -  0.29 

0.70 

0.69 

0.13 

0.30 

Table  4.6  shows  the  correlation  between  critical-thinking  skills  and  aca¬ 
demic  measures  of  success  as  reported  by  Insight  Assessment  [Insight  Assess¬ 
ment  2007]  and  by  this  study.  Both  results  show  a  positive  correlation  between 
SAT  verbal  and  math  scores  and  a  low  correlation  between  college  GPA  and 
critical-thinking  scores,  showing  that  there  is  not  a  strong  link  between  GPA 
and  critical-thinking  ability. 

4.2.3  Comparison  of  Experts’  and  Students’  Critical- 
Thinking  Scores 

In  response  to  Research  Question  6  concerning  the  comparison  of  the  critical- 
thinking  skills  of  professionals  with  the  scores  of  undergraduate  computer 
science  students,  A  NOVA  tests  and  Hsu’s  Multiple  Comparisons  determined 
differences  between  the  populations,  by  comparing  each  mean  with  the  best 
(largest),  as  shown  in  Figure  4.12.  The  analysis  shows  a  significant  difference 
between  the  scores  of  the  Air  Force  and  those  of  the  Army  and  Freshmen; 
however,  no  statistical  difference  showed  between  the  scores  of  senior,  faculty, 
industry,  and  Air  Force  participants. 

4.2.4  Additional  Findings 

Figure  4.13  shows  ANOVA  and  Hsu’s  comparisons  of  the  data  categorized  by 
ethnicity.  The  majority  of  the  participants  were  Caucasian,  and  the  sample 
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One-way  ANOVA:  TOTAL  versus  GROUP 


Source 

DF 

SS 

HS 

F 

P 

GROUP 

S 

2692.3 

538. 5 

23.63 

0.000 

Etc  or 

173 

3942  *  5 

22.8 

Total 

178 
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Army 
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, — . — j 
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16.0  20.0  24.0  28.0 


Hsu's  HCB  (Multiple  Comparisons  tilth  the  Best) 
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-1,278 
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-2,785 

0,456 

Amy 

-14.028 

-11,136 
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Figure  4.12:  Comparison  of  total  scores  for  all  populations 


Table  4.7:  Mean  total  scores  on  CCTST  for  all  participants 


GROUP 

n 

Mean 

StDev 

Var 

Min 

Max 

Freshmen 

54 

20.98 

5.02 

25.22 

6 

30 

Senior 

29 

24.93 

4.87 

23.71 

13 

33 

Faculty 

20 

26.10 

4,19 

17.57 

17 

32 

Industry 

18 

25.28 

3.27 

10.68 

20 

31 

Air  Force 

27 

26.56 

4.00 

16.03 

12 

31 

Army 

31 

15.42 

5.80 

33.65 

7 

28 
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Table  4.8:  Base  line  results  in  the  sub-categories  for  all  participants 


Variable 

n 

Mean 

StDev 

Var 

Min 

Med 

Max 

Induction 

179 

11.888 

3.037 

9.223 

3 

12 

17 

Deduction 

179 

10.603 

3.634 

13.207 

2 

11 

17 

Analysis 

179 

5.263 

1.478 

2.183 

1 

6 

8 

Inference 

179 

10.648 

3.245 

10.533 

2 

11 

16 

Evaluation 

179 

6.609 

2.406 

5.790 

0 

7 

11 

TOTAL 

179 

22.503 

6.105 

37.274 

6 

24 

33 

Table  4.9:  Two-tailed  T  test  of  women’s  scores  and  men’s 


GROUP 

N 

Mean 

StDev 

SE  Mean 

Women 

32 

21.75 

6.02 

1.11 

Men 

147 

22.67 

6.37 

0.51 

Difference  =  fi  (Women)  -  fi  (Men) 

Estimate  for  difference:  -0.92 
95%  Cl  for  difference:  (-3.29,  1.46) 

T-Test  of  diff=0:  T-VaJ  =  -0.78  P-VaJ  =  0.441  DF  =  46 


sizes  for  the  other  groups  were  too  small  to  draw  any  statistical  conclusions. 
Table  4.9  shows  no  statistical  difference  between  men’s  and  women’s  scores 
(P- value— 0.441,  DF— 46). 

In  Figure  4.14.  AN  OVA  and  Hsu’s  comparisons  of  total  scores  based  on 
the  main  area  of  the  participants  show  a  statistically  significant  difference 
in  the  scores  of  the  STEM  disciplines  when  compared  to  the  scores  of  other 
participants;  however,  senior  STEM  scores  showed  no  difference  from  those  of 
the  STEM  professionals. 

in  Figure  4.15,  ANOVA  and  Hsu’s  comparison  tests  of  total  scores  based 
on  education  levels  of  participants  show  a  significant  difference  between  the 
scores  of  the  participants  with  the  highest  mean  average  (those  with  PhDs)  and 
those  with  bachelor  degrees  and  the  freshmen.  No  statistical  difference  showed 
between  the  scores  of  those  with  PhDs  and  the  scores  of  senior  participants. 
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On# -way  AN  OVA:  TOTAL  vtr*us  Ethnicity 


Source 

OF  35 

H3 

r 

P 
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5  1073.7 
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Error 
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— - 1 - ———4 - + - 

14,0  17,5  21,0  24.5 
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Family  error  rate  -  0.05 
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Intervals  for  level 
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Center  Upper 
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-7.457  0,000 

( - - — * - 

-) 
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-11*206 

-0*124  0.000 

( - f — - . 
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-13.406 

-0,291  0.000  ( 

1  Choose  not 

-0.791 
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t - * . 

— 3 
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-5,124  0.000 

( - ■— - 

Caucasian 

0.000 
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(- 

-12,0  -6,0 


0,0  6.0 


Figure  4.13:  Comparison  of  mean  total  score  and  ethnicity 


On**way  AN  OVA:  TOTAL  v*rsus  MAIN  AREA 


Source 

DF 

S3 

MS 

F 

P 

HAIM  kPEk 

4 

2273,6 

568,4 

22,68 

0,000 
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174 

4361.2 

25-1 

Total 

178 

6634.7 

3  -  5.006  R-Sq  -  34.27%  R-Sq(adJ)  -  32.76% 
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[ — • - ) 
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50 
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5,890 

t- — *- — > 

Fresh  STEH 

24 

22.500 

4.  520 

t . * . j 

Senior  STEM 

24 

25.708 

4,  515 

. . . 

STEH 

58 

26.276 

3,712 

t — * — i 

18.0  21.0  24.0  27.0 


Hsu's  HCB  (Multiple  Comparisons  with  the  Best) 

Intervals  Cot  level  aean  Minus  largest  oC  other  level  Means 


Leva! 

Lower 

Center 

Upper 

Beh  Sci/Han 

-10,879 

-8.145 

0.000 

Other 

-9,863 

-7,676 

0.000 

Fresh  STEM 

-6,469 

-3,776 

0,000 

Senior  STEH 

-3,260 
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( . * . — ) 

( . * . . ) 

I . •- . > 

t . * . ) 

. . . 

-8.0  -4.0  0.0  4.0 


Figure  4.14:  Comparison  of  mean  total  scores  of  participants’  main  areas 
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One-way  ANOVA:  TOTAL  versus  Education  level 


Source 

DF 

SS 

MS  F  P 

Education 

level  7 

1045. 7 

149.4  4.57  0.000 

Error 
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5589.0 
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54 
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32 
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5.396 

. . . 
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28 
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6.532 

PhD 

21 

26,429 

3.400 

t — *. 

- + - + - - +- - 

7.0  14.0  21.0  29.0 
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Figure  4.15:  Comparison  of  mean  total  scores  based  on  education  level 
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4.3  Summary 

This  data  shows  that,  expert  computer  science  professionals  have  exceptional 
critical-thinking  abilities  and  scored  well  above  the  national  average.  The 
freshmen  and  the  seniors  also  scored  above  the  national  average,  with  the 
seniors  scoring  statistically  higher  than  the  freshmen.  The  experts  scored 
significantly  higher  than  the  freshmen;  although,  there  was  no  statistical  dif¬ 
ference  between  the  scores  of  the  experts  and  the  seniors.  Although  a  positive 
correlation  was  found  between  critical-thinking  and  SAT  scores  for  both  stu¬ 
dent  populations,  no  correlation  showed  between  critical-thinking  scores  and 
grade-point  average. 
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Chapter  5 

Conclusions  and  Recommendations 


A  “quiet  crisis”  is  growing  in  America  because  of  the  discrepancy  between 
the  increasing  need  for  top-tier  performers  in  computer  science  and  the  declin¬ 
ing  number  of  college  graduates  available  to  fill  these  positions.  In  addition  to 
the  need  for  more  computer  science  graduates,  these  students  must  be  equipped 
with  exceptional  critical-thinking  abilities,  beyond  subject  matter  content  ,  ex¬ 
emplified  by  top-tier  performers  in  the  industry.  This  study  addressed  that 
growing  concern  by  assessing  and  comparing  the  critical-thinking  abilities  of 
top-tier  computer  science  professionals  and  college  freshmen  and  seniors  in  the 
STEM  disciplines. 

Section  5.1  provides  a  summary  of  the  research  including  the  problem  ad¬ 
dressed,  the  background  literature  and  the  findings  of  the  study.  Section  5.2 
draws  conclusions  based  on  the  findings  of  the  research  questions  followed  by 
a  discussion  of  them  in  Section  5.3.  Section  5.4  concludes  the  chapter  with 
recommendations  for  practice  and  for  future  areas  of  research. 


5.1  Study  Summary 

Highly  skilled  top-tier  performers  in  computer  science  are  essential  to  Amer¬ 
ica’s  business  success  and  are  at  the  core  of  America’s  technological  edge,  na¬ 
tional  competitiveness  and  security.  While  the  need  for  these  skilled  computer 
science  professionals  is  increasing,  the  supply  of  potential  computer  science 
experts  is  declining.  Adding  complexity  to  this  issue,  many  companies  have 
found  that  subject  matter  knowledge  and  academic  record  are  not  necessar- 


ily  good  indicators  of  the  traits  that  enable  newly  hired  employees  to  become 
top-tier  performers.  Research  in  expertise  also  supports  this  conclusion,  find¬ 
ing  that  expert  performers  have  cognitive  skills  that  are  more  likely  to  predict 
success  than  do  traditional  measures  of  academic  merit.  Additionally,  recent 
guidelines  of  the  computer  science  accreditation  board  imply  the  need  to  in¬ 
corporate  critical-thinking  skills  into  the  curriculum  [ABET  2007]. 

The  purpose  of  this  study  was  to  assess  the  critical- thinking  abilities  of 
top- tier  computer  science  performers  from  college  faculties,  industry  and  the 
military,  and  of  undergraduate  freshmen  and  seniors  in  the  STEM  disciplines. 
The  core  hypothesis  is  that  1)  top-tier  performers  in  computer  science  have 
highly  developed  critical-thinking  abilities  and  2)  that  college  students  in  the 
STEM  disciplines  lack  these  abilities.  The  data  supports  the  first  element  of 
the  hypothesis,  but  it  does  not  support  the  second.  Guiding  the  research  to 
establish  this  argument  were  the  following  areas  of  inquiry: 

•  The  critical- thinking  abilities  of  top-tier  professionals 

•  The  critical- thinking  abilities  of  freshmen  students 

•  The  critical-thinking  abilities  of  senior  computer  science  students 

•  The  comparison  of  the  critical-thinking  skills  of  college  freshmen  with 
those  of  senior  computer  science  students 

•  The  comparison  between  currently  used  measures  of  academic  perfor¬ 
mance,  such  as  grade  point  average,  and  critical-thinking  skills 

•  The  differences  in  critical-thinking  skills  between  computer  science  pro¬ 
fessionals  and  the  student  populations  tested 

To  assess  the  critical-thinking  abilities  of  volunteers  from  different  popula¬ 
tions,  this  study  used  the  California  Critical  Thinking  Skills  Test  (CCTST). 


71 


An  analysis  of  the  data  collected  from  these  assessments  shows  the  following: 

•  The  mean  total  critical-thinking  score  on  the  CCTST  of  expert  popula¬ 
tions  from  academia,  industry  and  the  military  is  significantly  above  the 
national  average. 

•  Both  student  populations  scored  higher  than  the  national  average. 

•  The  senior  computer  science  students  scored  significantly  higher  than 
the  freshmen  students. 

•  Although  the  analysis  revealed  a  positive  correlation  between  the  stu¬ 
dents’  scores  on  the  verbal  and  the  math  SAT’s,  there  was  little  correla¬ 
tion  found  between  critical-thinking  abilities  and  grade  point  average. 

•  The  seniors  scored  as  well  as  those  in  the  expert  populations  with  the 
exception  of  the  Air  Force,  with  outlying  scores  removed,  who  scored 
significantly  higher  than  the  seniors. 

5.2  Conclusions 

Data  presented  in  Chapter  4  shows  that  highly  developed  critical-thinking 
skills  are  characteristic  of  expert  top-tier  performers  in  computer  science.  It 
also  shows  that  these  experts  score  significantly  higher  on  critical- thinking 
abilities,  as  measured  by  the  CCTST,  than  do  the  freshmen,  suggesting  that 
critical  thinking  is  a  learned  skill. 

The  current  study  also  shows  that  the  seniors  do  have  high  levels  of  critical- 
thinking,  as  measured  by  the  CCTST;  in  fact  on  a  statistical  level  equivalent  to 
the  defined  “experts.”  At  the  same  time  the  seniors  scored  significantly  higher 
than  the  freshmen,  which  on  the  surface  at  least,  seems  to  demonstrate  a  pro¬ 
gression  of  critical- thinking  abilities  -  suggesting  the  undergraduate  curriculum 
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Critical  Thinking  Total 


— Average 


Figure  5.1:  Critical  thinking  scores  of  the  sample  populations  tested  showing 
a  significant  difference  between  the  scores  of  the  freshmen  and  those  of  the 
seniors  and  experts  and  no  statistical  difference  between  the  scores  of  the 
seniors  and  those  of  the  experts. 


does  have  an  impact  on  the  development  of  critical-thinking  skills  of  computer 
science  students.  However,  how  and  when  these  skills  are  developed  is  unclear 
from  this  work  and  remains  an  area  for  future  research.  Figure  5.1  shows  the 
progression  of  these  scores.  Intermediate  scores  between  freshmen  and  senior 
levels  were  not  assessed,  and  the  uncertainty  of  the  shape  and  nature  of  the 
change  that  occurs  during  an  undergraduate  computer  science  education  raises 
many  questions  for  future  research. 

Industry  research  indicates  that  students  with  high  grade-point  averages 
do  not.  necessarily  have  high  levels  of  critical-thinking  ability  and  that  students 
who  score  low  on  tests  and  projects  may,  in  fact,  have  high  critical-thinking 
abilities  [Colwell  2005].  This  earlier  finding  is  supported  by  the  current  study 
in  that  the  data  shows  little  relationship  between  grade-point  average,  the 
assumed  measure  of  academic  success,  and  critical-thinking  abilities. 
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5.3  Discussion 


Unlike  past  studies  that  focused  primarily  on  the  academic  success  of  un¬ 
dergraduates  to  predict  expert  performance,  this  research  took  a  different  ap¬ 
proach  by  identifying  the  presence  of  critical-thinking  skills  of  top-tier  perform¬ 
ers,  those  in  the  computer  science  profession  with  5  to  10  years  of  experience. 
Their  critical- thinking  scores  can  serve  as  a  benchmark  of  critical- thinking 
skills,  red  a  standard  that  can  be  compared  to  the  scores  of  midergraduate 
students,  and  can  be  used  to  guide  the  development  of  curriculum  and  course 
modules  that  stress  the  development  of  these  cognitive  skills, 

The  current  work  demonstrates  that  critical-thinking  abilities  of  computer 
science  students  improve  from  the  freshmen  to  the  senior  year,  that  is  those 
who  choose  computer  science  as  a  major  and  survive  to  graduation  do  gain 
some  measure  of  critical-thinking  skills.  How  such  enhancement  occurs  and 
when  it  happens  is  unclear  but  suggests  such  questions  as:  Do  those  with  high 
levels  of  critical-thinking  abilities  survive  in  the  computer  science  discipline,  in¬ 
dicating  a  need  for  this  skill  when  entering  as  freshmen,  or  do  students  develop 
critical-thinking  skills  while  in  the  computer  science  discipline?  A  longitudinal 
study  measuring  critical-thinking  abilities  throughout  an  undergraduate  career 
would  help  to  answer  these  questions. 

It  is  generally  accepted  that  the  needs  of  the  computer  science  industry 
require  that  the  computer  science  curriculum  have  a  particular  interest  in  and 
emphasis  on  the  development  of  these  skills;  however,  this  current  research  does 
not  investigate  how  critical-thinking  skills  are  enhanced  and  what  experiences 
create  the  atmosphere  for  improving  these  abilities  at  the  undergraduate  level. 
Significant  questions  remain  unanswered,  such  as,  “What  individual  factors 
affect  critical-thinking  scores?”  “Do  critical-thinking  skills  improve  incremen¬ 
tally,  or  is  there  a  point  at  which  the  students  ‘get  it’  and  their  skills  improve 
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Figure  5.2:  Possible  shapes  of  the  curve  depicting  the  change  in  critical- 
thinking  abilities  from  freshmen  to  senior  computer  science  students,  as  mea¬ 
sured  by  the  CCTST. 

dramatically?”  “Do  all  disciplines  see  the  same  types  of  progression  of  critical- 
thinking  abilities?”  Figure  5.2  shows  the  possible  shapes  of  the  curve  depicting 
the  change  in  critical-thinking  abilities  from  freshmen  to  senior  computer  sci¬ 
ence  students  as  measured  by  the  CCTST.  These  and  other  questions  remain 
unanswered  and  are  subjects  for  future  study. 

Importantly,  the  current  study  shows  that  critical-thinking  skills  have  little 
correlation  with  grade-point  average,  so  it  is  reasonable  to  ask  if  these'  skills  are 
being  sufficiently  measured.  The  importance  of  developing  exceptional  critical- 
thinking  skills  make  their  assessment  essential  for  computer  science  students. 
Students  who  rank  lower  academically  and  have  a  lower  CPA  may  still  be 
learning  these  important  critical-th inking  skills  without  their  being  recognized 
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and  nurtured.  Measuring  academic  achievement  with  content-oriented  tests  is 
important  and  represents  the  core  of  computer  science  knowledge.  However, 
this  study  has  shown  that  GPA  does  not  accurately  reflect  critical-thinking 
abilities  and  suggests  that  assessments  need  to  bo  developed  and  implemented 
to  assess  these  skills. 

Another  interesting  issue  concerning  the  student  population  is  that  its 
make-up  may  have  possibly  influenced  its  mean  score  on  the  CCTST.  Roughly 
half  of  the  freshmen  had  declared,  or  intended  to  declare,  STEM  discipline  ma¬ 
jors,  which  possibly  accounted  for  the  higher  critical-thinking  scores  for  those 
participants.  The  remaining  freshman  participants  had  selected  a  wide  scope 
of  majors;  therefore,  their  scores  helped  to  generalize  the  freshmen  student 
population.  Most  of  the  senior  participants  came  from  a  compiler  construc¬ 
tion  course,  a  capstone  course  designed  to  challenge  the  students  to  use  the 
breadth  of  their  knowledge.  The  other  senior  students  also  came  from  STEM 
disciplines.  A  result  is  that  the  freshmen  data  tended  to  be  more  general¬ 
ized  than  that  of  the  senior  group  and  suggests  that  additional  work  across  a 
broader  spectrum  of  majors  could  be  done  to  good  effect.  A  university-wide 
approach  could  be  especially  revealing,  allowing  the  researcher  an  opportunity 
to  consolidate  up  or  down  as  the  data  leads. 

Surprising  results  came  from  the  United  States  Army  Signal  Corps  Offi¬ 
cers’  scores.  Initially,  these  participants  were  thought  to  be  a  military  expert 
population,  similar  to  the  Air  Force  group.  However,  they  scored  relatively  low 
on  all  phases  of  critical  thinking.  In  an  effort  to  explain  the  make-up  of  the 
group,  the  commandant  of  the  school  informed  the  researcher  that  they  art* 
not  considered  computer  experts  but  are  officers  who  represent  a  wide  breadth 
of  professions  and  experience.  They  are  not  necessarily  top-tier  performers, 
and  the  nature  of  their  mission  is  quite  different  from  that  of  the  Air  Force 
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population.  They  better  represent  non- technical  professionals  and  serve  as  an 
interesting  comparison  with  the  other  professional  populations. 

Interesting  results  also  came  from  the  scores  of  participants  from  one  par¬ 
ticular  company.  The  contact  selected  a  subjective  group  of  participants  based 
on  their  performance  including  groups  at  three  performing  levels  he  called  “A” 
or  high-level,  “B”  or  middle-level,  and  “C”  or  low-level.  Although  the  sample 
size  was  too  small  to  make  any  statistical  conclusions,  the  scores  of  the  “A” 
level  performers  were  higher  than  those  of  the  “B”  and  “C”  levels  with  those 
of  the  “C”  level  scoring  the  lowest.  This  suggests  that  the  expert  performers, 
those  who  were  rated  highest  by  the  supervisors,  had  higher  critical- thinking 
abilities,  as  measured  by  the  CCTST,  than  did  the  average  or  lower  level 
performers  within  that  company.  Further  research  is  needed  to  compare  the 
critical-thinking  abilities  of  the  top-tier  performers  with  those  of  the  middle 
and  lower-level  performers  in  various  companies. 

Limitations  of  Current  Study  In  assessing  the  critical-thinking  abilities 
of  participants  and  drawing  conclusions,  certain  limitations  must  be  consid¬ 
ered.  The  assessment  used  in  this  study  is  a  one-time  test  to  measure  the 
critical-thinking  skills  of  the  participants.  Many  factors  can  possibly  influence 
a  participant’s  score  on  any  particular  test  including,  among  others,  the  per¬ 
son’s  mood  at  the  time  of  the  test,  distractions  in  the  testing  environment,  and 
the  degree  of  concentration  and  effort  on  the  part  of  the  participant.  These  can 
certainly  be  limitations,  but  are  uncontrollable  with  a  test  such  as  the  CCTST; 
therefore,  critical-thinking  skills  need  to  be  assessed  and  observed  several  times 
over  a  4- year  education  to  ensure  that  the  skills  are  being  learned. 

This  study  was  limited  to  the  CCTST  that  measures  certain  aspects  of 
critical-thinking  by  assessing  the  cognitive  abilities  of  the  participants.  Other 
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validated  critical-thinking  tests  should  be  used  with  their  results  compared 
with  the  results  of  the  CCTST  used  in  this  study.  Selecting  a  test  with  a  greater 
number  of  questions  may  improve  the  spread  of  the  scores;  however,  a  longer 
test  would  require  greater  time  commitment  on  the  part  of  the  volunteers  and 
make  it  more  difficult  to  get  participants. 

Another  potential  limitation  is  the  sample  size  of  the  populations  mea¬ 
sured.  Although  the  groups  used  in  this  study  were  large  enough  from  which 
to  draw  statistical  conclusions  and  larger  than  most  in  previous  similar  stud¬ 
ies,  a  greater  number  of  participants  would  produce  a  broader  base  of  data 
and  would  provide  the  researcher  with  more  flexibility  for  grouping  and  con¬ 
solidating  results  across  additional  criteria.  Also,  this  assessment  was  limited 
to  freshmen  and  senior  students  at.  one  major  undergraduate  institution.  Ex¬ 
panding  this  research  across  universities,  both  public  and  private,  and  to  all 
class  years  would  provide  valuable  information. 

Previous  research  has  found  that  critical- thinking,  problem-solving,  and 
creativity  are  all  components  of  the  cognitive  thinking  skills  contributing  to 
expert  performance.  This  study  focused  solely  on  critical-thinking  skills  and 
no  attempt  was  made  to  investigate  the  overlap  that  could  exist  among  the 
three  individual  skills.  Because  problem-solving  and  creativity  skills  can  pos¬ 
sibly  influence  critical-thinking  abilities,  future  research  needs  to  explore  this 
connection  as  it  relates  to  computer  science, 

5.4  Recommendations 

5.4.1  Recommendations  for  Future  Study 

Research  into  the  cognitive  abilities  of  experts  is  an  emerging  Held.  Future  re¬ 
searchers  interested  in  pursuing  studies  pertaining  to  the  roles  of  the  cognitive 
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abilities  of  expert  computer  science  performers,  to  improving  the  undergrad¬ 


uate  curricula  and  to  additional  related  topics  can  find  a  wide  array  of  them 
from  which  to  choose.  For  instance, 

•  An  assessment  of  critical  thinking  across  a  wide  range  of  top- tier  ex¬ 
perts  from  a  variety  of  computer  professions;  including  software  engi¬ 
neers,  architects,  graphic  designers  and  other  specialties;  could  provide 
a  broad-base  benchmark  for  computer  skills. 

•  An  assessment  of  critical-thinking  scores  of  expert  populations  outside 
the  computer  science  profession  could  serve  as  an  interesting  measure  for 
comparison  and  contrast  of  cognitive- thinking  abilities. 

•  Conducting  a  longitudinal  study  of  a  particular  sample  population  through¬ 
out  a  4- year  college  education  could  provide  insights  into  the  changes  in 
critical-thinking  skills,  show  the  progression  of  these  skills,  and  allow 
for  the  measurement  of  small  changes  taking  place  in  these  cognitive- 
thinking  abilities.  Careful  selection  of  an  appropriate  assessment  tool  is 
essential  for  a  long-term  research  project  such  as  this  that  has  a  large 
test-bank  of  questions.  A  longitudinal  study  could  provide  insights  to 
the  following  questions: 

-  Is  there  a  common  point  in  the  computer  science  curriculum  at 
which  students  make  the  most  improvement  in  critical-thinking  abil¬ 
ities? 

-  Is  there  a  particular  course  or  course  sequence  that  causes  the  most 
gains  in  critical- thinking  abilities? 

-  Is  there  a  particular  point  at  which  students  seem  to  grasp  critical- 
thinking  concepts  and  make  large  improvements  in  a  short  period 
of  time? 
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-  How  do  courses  designed  to  promote  problem-solving  skills,  such 
as  those  employing  problem- based  learning  models,  affect  critical- 
thinking  skills? 

-  How  does  attrition  rate  of  students  leaving  the  computer  science 
major  relate  to  critical- thinking  skills? 

-  Are  those  students  that  survive  in  computer  science  the  ones  with 
higher  initial  critical-thinking  abilities,  or  is  this  a  learned  skill? 

—  How  do  different  schools  and  curricula  affect  the  learning  curve  of 
critical- thinking  abilities? 

-  Are  there  other  factors,  such  as  gender,  age,  geographic  region,  or 
ethnicity  that  affect  the  learning  of  critical- thinking  skills? 

This  longitudinal  study  could  be  expanded  to  follow  students  after  grad¬ 
uation  measuring  changes  in  cognitive  abilities  throughout  their  profes¬ 
sional  career,  providing  answers  to  the  following: 

-  Do  particular  career  patios  promote  and  enhance  critical- thinking 
abilities  more  than  others? 

—  Do  critical-thinking  abilities  decline  or  improve  over  time  in  com¬ 
puter  professions  in  general? 

-  Do  critical-thinking  abilities  predict  or  correlate  to  success  in  a  com¬ 
pany,  as  defined  by  position  or  responsibility? 

•  Studies  of  the  critical-thinking  abilities  of  students  from  a  large  range  of 
populations  such  as  different  disciplines  and  various  age  groups  can  be 
compared  and  contrasted  with  the  scores  of  computer  science  students. 

•  Students  from  different  institutions-private  colleges,  community  colleges. 
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small  schools  and  major  universities-could  be  assessed  to  measure  dif¬ 
ferent.  factors  that  may  influence  critical-thinking  abilities. 

•  An  assessment  of  the  critical- thinking  abilities  of  incoming  freshmen  and 
how  these  initial  scores  affect  and  predict  success  at  the  undergraduate 
level  could  be  conducted  in  different  disciplines. 

•  The  assessment  of  the  creativity  component  of  expert  performance  and 
its  correlation  to  expertise  and  to  critical-thinking  abilities  could  be  stud¬ 
ied. 

In  short,  a  wider  study  of  the  impact  of  critical  thinking  on  the  higher 
education  experience  as  a  whole  and  its  relationship,  if  any.  to  student  success, 
including  when  and  how  these  changes  are  manifested  in  students,  needs  to  be 
conducted.  The  current  study  is  a  first  step  toward  recognizing  the  importance 
of  these  cognitive  skills  beyond  subject-matter  knowledge. 

5.4.2  Curriculum  Issues 

One  of  the  key  issues  facing  computer  science  education  researchers  is  that 
“colleagues  regularly  convey  the  attitude  that  educational  research  is  not  real 
research”  [Alstrum  et  al.  2005].  Rather  than  being  discouraged  from  accept¬ 
ing,  participating  in,  and  benefiting  from  the  results  of  this  type  of  research, 
institutions  and  their  faculties  must  be  educated,  encouraged  and  helped  to 
appreciate  and  understand  the  process  and  of  its  benefits  for  them  and  their 
students.  The  following  recommendations  are  intended  to  help  institutions 
and  their  faculties,  and  ultimately  the  students,  to  get  the  most  benefit  from 
this  type  of  research. 
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•  The  importance  of  critical-thinking  skills  should  be  recognized  and  ways 
to  incorporate  the  skills  into  all  levels  of  the  undergraduate  education 
should  be  explored. 

•  A  repository  of  critical- thinking  exercises  and  details  on  how  to  integrate 
them  into  existing  courses,  along  with  sample  lesson  plans  containing  ac¬ 
tivities  and  assignments,  should  be  developed  to  assist  computer  science 
educators  in  promoting  and  fostering  these  cognitive  skills. 

•  Critical- thinking  skills  should  be  evaluated  throughout  the  educational 
process  with  assessments  made  at  key  points  throughout  the  curriculum 
to  continually  measure  progress. 

•  A  variety  of  critical-thinking  assessment  tools  must  be  made  available  to 
all  instructors  for  integration  into  key  computer  science  courses. 

•  Computer  science  faculty  particularly  should  embrace  new  educational 
research  and  recognize  its  importance  in  shaping  the  scope  of  educational 
programs. 

•  Preparing  students  to  be  top-tier  performers  with  the  tools  and  abilities 
beyond  subject-matter  expertise  must  be  a  goal  of  all  undergraduate 
institutions. 

The  lessons  learned  and  the  information  obtained  as  a  result  of  this  study 
can  be  invaluable  to  future  researchers,  to  institutions  with  computer  science 
majors  and  ultimately  to  the  computer  science  industry  as  a  whole. 
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APPENDICES 


A  Assessments  of  Creativity 

See  the  Indiana  University  website, 

http:// www,  indiana.edu/bobweb/Handout/cretv_6.html, 

for  further  information  on  the  following  creativity  assessments. 

•  Divergent  Thinking  Tests 

-  Guilford’s  Alternate  Use  Task  1967 

-  Wallas  and  Kogan  1965 

-  Torrance  Tests  of  Creative  Thinking  1974 

•  Convergent  Thinking  Tests 

-  Insight  Questions 

-  Mednick’s  Remote  Association  Task  1962 

•  Artistic  Assessments 

-  Barron- Welsh  Art  Scale 

•  Self  assessments 

-  Khatena- Torrance  Creative  Perception  Inventory 
—  How  Do  You  Think  (Davis) 

-  Things  Done  on  Your  Own  (Torrance,  1962) 

—  The  Creativity  Behavior  Inventory 

-  Runco  Ideation  Behavior  Scale  (RIBS) 

—  Creative  Attitude  Survey  (Schaeffer) 
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-  Statement  of  Past  Activities 


-  NEO-PI-R  (Openness  to  Experience  component) 

-  Gough  Personality  Scale 

•  Other  Assessments 

-  Creativity  Assessment  Packet 

-  Preschool  and  Kindergarten  Interests  Descriptors 

-  Scales  for  Rating  the  Behavioral  Characteristics  of  Superior  Stu¬ 
dents  (Renzulli,  1993) 


87 


B  Assessments  of  Critical  Thinking 

The  following  is  a  brief  list  of  some  of  the  most  popular  critical  thinking  tests. 
Project  CAT  Tennessee  Tech 

“Project  CAT  is  a  cooperative  project  sponsored  by  the  National  Science  Foun¬ 
dation  and  Tennessee  Technological  University.  The  project  goal  is  to  refine 
an  instrument  for  assessing  critical  thinking  skills  in  undergraduate  students” 
[Stein  2006]. 

The  following  abbreviated  list  of  assessments  of  critical  thinking  comes 
from  Dr.  Ennis  at  the  University  of  Illinois 
( http:  /  /faculty,  ed .  uiuc .  edu/rhennis/testlistrevised606 .  lit  m) : 

•  “California  Critical  Thinking  Dispositions  Inventory”  (1992)  by  Peter 
Facione  and  N.  C.  Facione. 

•  “Cornell  Critical  Thinking  Test  -  X”  (2005)  by  Robert  H.  Ennis  and 
Jason  Mi  liman. 

•  “Cornell  Critical  Thinking  Test  -  Z”  (2005)  by  Robert  H.  Ennis  and  Jason 
Millman, 

•  “Critical  Thinking  Interview”  (1988)  Gail  Hughes  and  Associates. 

•  “The  Collegiate  Assessment  of  Academic  Proficiency  (CAAP)  Critical 
Thinking  Test  Distributed  by  ACT” 

•  “Ennis- Weir  Critical  Thinking  Essay  Test”  (1985)  by  Robert  H.  Ennis 
and  Eric  Weir. 

•  “International  Center  for  the  Assessment  of  Thinking”  (1996). 

•  “James  Madison  Test  of  Critical  Thinking” 
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•  “New  Jersey  Test  of  Reasoning  Skills”  (1983)  by  Virginia  Shipman. 

•  “ETS  Tasks  in  Critical  Thinking”  (1993). 

•  “Watson-Glaser  Critical  Thinking  Appraisal”  (1980)  by  Goodwin  Watson 
and  Edward  Maynard  Glaser). 
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C  California  Critical  Thinking  Test 


The  California  Critical  Thinking  Test  {CCTST),  developed  by  Peter  Fracione 
[Facione  1990],  is  a  34-question,  multiple-choice,  on-line  assessment  that  mea¬ 
sures  5  areas  of  critical  thinking  (inductive  reasoning,  deductive  reasoning, 
analysis,  inference,  evaluation,  and  total  critical  thinking  score)  and  provides 
the  test  taker  immediate  feedback  on  the  results  (www.insightassessraent.com). 


The  CaSfomia  Critical  Thinking  SkSs  Test  -  CCTST 


Lead  Author:  Peter  A-  factor!*,  Ph.D. 

t  tWC  Form  A.  tftt?  FOrrt  B  »M  2W0 


Referred  to  by  some  as  "The  Gold  Standard"  of  critical  thinking  skills  tests... 


Cwitlmct  #«d  Canttnl  VtUityi  Th«  CCTST  is  bawd  qn  tbm  cfmttptuAliiilfon  td  thinking  HhtultUd  In  th*  IxpAd 

Con«*nHJj  itatlWflt  ewi  Coll*D*  L*v*l  €***»!  Thinking  (il»)  kMM  tf 


thm  Pulahi  Thii  tonctflt  n*i  tuOptuttd  by  *0  indip«ndant  mplfeattoh  tMUKh  atudy  of  pobey-mak***. 

■mpiayar*,  *nd  whek  hm  conducted  *1  Pton  £UU  Urdu*™**,  *t>vniqr«d  Iry  OS-  0*partrft*nt  of  EduC*b*n- 

S-CW**  Jteportedi  Th*  CCTST  Total  Scot*  targets  th*  sttengtti  or  Mvikrwn  of  on*'*  fkil(  in  making  r#fl*ct>v*.  r**cqn*0 
)udgm*nta  About  «ha!  to  ball*1*#  or  *dwt  to  do.  Th*  CCTST  gw*r«tll  i*v*r*J  KOTO*  rtlitin^  to  thinking. 


*  Cv«'«ll  critical  (funking  Tkjffft  total  vcor*  and  norm-group  (mrcarrhl*, 

*  Siti-k*!*  t<wn ■  fyt  th*  <l*l*iC*J  *J  InducUu*  K*  atoning  mod  Oadiatnf  K*a»e**t»g 

*  atua  stem*  by  th*  cprtfamporary  «  of  Analys-i*,  inf  •rant*,  And  £  valuation 


d  CCT5T  Acak  rii 


1  nta«fed*d  SuttyacTs:  Th#  CCTST  It  d*«igh*d  for  coilAgt  ia<5«n(i  *oflj  adults  It  hit  b#*n  mo«t  te£Hdv  Vl*d  **Th  tradition*! 
*fl*d  cottag*  and  unlvwifty  undvgmduato  students,  Mttfi  students  ,n  prot*x*H?n*l  m**t#n  r**  programs,  and  wth 
ttmmuniit  ;o'i*p*  students  to  gather  dot*  cm  individual  and  group  trttkml  tanking  *M1  l*v*l*  ■■  ompt*d  to  Ipctlh 

()Av«!otwd  OT  I*  d***lop*d  nation*  I  norm*. 

t>*rpp***:  TH*  CCtJT  h*t  b**n  ua*4  ntliantlly  *nd  ini*rn*Uon*l ly  for  turning  Outturn**  lAMii^lhL  pAfforrntnt* 
funding,  program  evaluation.  pmfaaatonal  development  training,  and  a*  an  a  lament  in  application.  adrmaakona.  and 
panonnal  evaluation  protauta*. 

F«*mati  ail  forma  of  th*  COST  am  3*- item  multiple  dvo*c*  format 
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Given  the  importance  of  critical  thinking  to  our 
democracy*  our  economy,  and  our  Jives  m  a  pluralistic,  global  community* 
we  hope  that  you  get  most,  if  not  all,  of  these  nght. 

Click  on  the  link  after  each  item  to  see  an  analysis  of  that  item  and  its  various  choices. 

The  items  shown  here  are  similar  to  those  found  on  the  CCTST  and  the  I£g 


Sample  Reasoning  Skills  Item  #1:  Using  the  phone  at  her  desk  Syfvta  in  Corporate  Sales  consistently 
generates  a  very  steady  SI  §00  per  hour  in  gross  revenue  tor  her  firm  After  all  of  her  firms  costs  have  been 
subtracted  Syhias  sales  amount  to  $100  m  bottom  line  met  I  profits  wry  15  minutes  Ai  10  00  a  m  one  day 
the  desk  phone  Sytaa  uses  to  maker  her  sales  calls  breaks  Without  the  phone  Sylvia  cannot  make  any  sales 
Assume  that  Sylvias  regular  schedule  is  to  begin  making  sales  calls  at  8  00  a  m  Assume  she  works  the  phone 
for  tour  hours  takas  a  one  hour  lunch  •*# ctly  at  noon  and  then  letums  promptly  lo  her  desk  tor  lour  more  hours 
of  afternoon  sales  Sybil  loves  her  work  and  the  broken  phone  is  keeping  her  from  it  If  necessary  she  will  try  lo 
repair  the  phone  herself  Which  of  the  following  options  would  be  m  the  best  interest  of  SyMes  firm  to  remedy 
the  broken  phone  problem? 


A  -  Use  Eds  Phone  Repair  Shop  down  the  street  Ed  can  replace  Sybil's  phone  by 
10  30  a  m  Ed  will  charge  the  firm  55QG 

0  -  Assign  Sykia  to  a  different  project  until  her  phone  can  be  replaced  with  one  horn 
the  firm  s  current  inventory  Replacing  the  phone  is  handled  by  the  night  shift 

C  -  Authorize  Sylvia  to  buy  a  new  phone  during  her  lunch  hour  for  $75  knowing  she 
car  plug  it  in  and  have  it  working  within  a  few  minutes  after  she  gets  back  to 
her  desk  at  1  00  p  m 

0  *  Ask  Syhia  to  try  to  reparr  her  phone  herself  She  will  probably  complete  the 
repair  by  2  00  p  m  or  maybe  later 


z*7  *rt  K*  r  s*  hi-  »*  re/a  3tas*i 


Sample  Reasoning  Skills  Item  §2:  Iv*  heard  many  reasont  why  our  nation  should  reduce  its  reliance  on 
petroleum  vehicle  fuels  One  is  that  relying  on  imported  oil  mikes  our  economy  dependent  on  the  political  whims 
crl  foreign  rulers  Another  is  that  other  energy  sources  like  the  possibility  of  hydrogen  based  fuels  are  less 
harmful  to  the  environment  And  a  third  is  that  petroleum  is  not  a  renewable  resource  so  when  wtv*  used  it  all 
up  it  mil  be  gone1  But  \  doirt  think  were  iikeiy  to  use  it  ell  up  for  at  least  another  fifty  years  And  by  then  well 
have  invented  new  and  better  tools  and  more  toek  efficient  vehicles  too  So  that  argument  doesn  t  worry  me  And  I 
dent  really  belie.*  the  stuff  about  how  foreign  Leaders  can  force  our  nation  to  change  its  policies  simply  by 
decreasing  their  oil  production  Oil  companies  like  Exxon  have  made  record  profits  precisely  m  those  times 
when  the  supply  of  foreign  Oil  was  reduced  1  dent  see  the  b»g  oil  companies  being  very  interested  in  pokey 
change  when  the  money  is  rotlmg  in  And  tor  another  our  nation  has  demonstrated  that  A  is  willing  to  wage  war 
rather  than  to  permit  foreign  leaders  to  push  us  around  So  ihis  whole  thing  about  how  we  has*  to  reduce  our 
reliance  on  petroleum  based  gasoline  diesel  and  jet  toil  is  bogus  ’  The  speaker  s  reasoning  is  best  evaluated 


A  -  solid  frl  shows  the  arguments  for  reducing  petroleum  vehicle  fuels  are  weak 
B  -  sobd  The  speaker  rs  very  clear  about  what  he  believes  and  why  he  believes  rt 
C  -  weak  The  speaker  probably  owns  stock  in  Exxon  or  some  other  oil  company 
D  -  weak  The  speaker  ignored  the  environmental  argument  entirely 

aia  »*n  te*  «■*  i«sii~«ir  ■*  wtr  in  2su.:t 


Sample  Reasoning  Skills  Item  13:  Consider  the  derm  Even  the  General  occasionally  uses  svasr.* 
language  as  this  damn  relates  to  the  following  reason  After  all  most  politicians  stn.e  to  please  their  venous 
constituencies  And  the  General  although  a  wise  forthright  articulate  and  seasoned  leader  like  all  important 
people  has  to  be  something  of  a  politic  ran  in  order  to  be  successful  I  find  it  very  hard  to  imagine  always  being 
able  to  please  every  constituency  without  at  Least  on  some  occasions  using  evasive  language  Assuming  all 
the  statements  made  as  part  of  the  reason  are  true  the  initial  claim  about  the  General  using  evasive  language  at 
times 


A  -  could  not  be  false 
B  *  is  probably  true  but  may  be  false 
C  *  is  probably  falsa  but  may  be  true 
D  -  could  not  be  true 


SC007  7>«  LLC.  UiIIe*m  C* 
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The  Scales  scores  for  the  CCTST,  TER*  HSRT,  and  the  0CTST 


Scat*  Fnducttv*  R«a*ontng:  lndu:b«  raviontng  happ*n*  ^H*n  *m  d#cJd*  that  th*  *uld*nc*  at  hand  m**ni  that  a 
gis#n  conclusion  *t  pnobawy  bv*.  Fpr  axampl*  if  rm  know  that  th#  vast  majority  of  paopi#  Vvo  »mok(,  ■■ 
compared  to  tn«i#  ,vh©  do  not  imijM,  iuff*r  layout  health  pr0oi*ms,  «#  might  rttMiU^  tondud*  by  indv d>v* 
'4aiev<vg  that  amoving  14  probably  haiardous  to  on# s  h*alih.  £a#nt*fic  rtnowg  aims  to  show  that  soma  id#as 
TiQ^a  l«k#1y  to  d*  tfy#  than  othars .  fci*mttstt  ut*  mduttiv#  mvtncos  such  as  *xp*nm*ntationi  and  th*y  via 
tnductiv*  tool*  such  as  statistics  When  w*  bat#  Our  s^dirbont  about  how  thing  a  mill  happen  m  th#  future  on  our 
past  experiences  ar#  utrng  inductive  rationing  As  tong  at  there  t  a*#n  th*  most  r*mot#  and  osscura 
poi**bilitv  that  although  all  th*  raaiorva  for  a  claim  could  be  true  and  y*t  th*  claim  might  iWU  b*  fall*  w*  ar# 

n  th#  realm  of  inducblva  *#■  toning 


Seal#  0#duchv#  Reasoning*  Deducts#  reasoning  happens  when  wa  dead*  that,  no  matter  unat  it  is  irfipessiol#  that 
th*  condo  11  on  ar*  conj id* ring  is  1*1*#'  giy#n  that  all  th*  9*111111  o*  our  argument  ar#  try*  For  exams!*  if  -  * 
know  V  •  fact  that  tan  Ci*go  is  nest  of  D*n v*r,  ang  in*  <n*fi  that  P*ny*r  ■»  west  of  D«trg<t  and  h*w  vpi*  than  w* 
can  inf*r  with  d#duttiv#  certainty  that  fan  Ciago  n  matt  *■'  H*w  York.  Mathematics  um  deductive  reasoning.  algebra 
and  gaometr,  am  ex*/(it«a  in  deduct!*#  reasoning,  Waving  «  gam*  can  also  b*  *n  #**rtlt*  W  deduct!*#  reasoning 
and  10  can  filling  out  an  intom*  tax  return.  For  both  gama*  and  lax  r#tums  ar*  things  that  r*quie*  us  to  asolv  stnet 
rules  and  lay#.  For  exempt#,  M  th#  batter  swings  and  misa«t  tire*  pptfnei,  th#  batt#r  1  out  and  Johnn.#  just  did 
that  10  John ph*  11  out'  1*  a  d«duetlv#  inf#r*nc«-  On#  of  th*  **ys  that  w*  know  that  littl#  children  can  '*nei 
deduct**#),-  1*  to  sbtfhi  that  they  can  play  gm«i  that  ncvi*  following  rules  *wn  playground  ul*a 


Seal*  AnalyifCi  W#  1*  using  our  analytical  sktNs  whan  w#  pull  apart  rgviunti  and  pomts  o'  v+*w  to  show  why  a 
panon  think*  what  n#  or  in*  thinks  In  effect  *r*  separating  th*  p<emu*f  and  th*  aaiumpliom  a  pmen  n 
wiphg  from  th*  diim  or  th#  conclusion  that  tin*  parson  it  machtng  For  *xa mp4*.  *uo»t*  *om*on*  propose*  tnat 
that  «e  should  go  to  w*r  oacaui#  th*  *n#my  is  building  up  weapons  pf  mats  oettruction  to  use  aga-nst  us  *n 
analy *>s  or  th!  paraon  *  pontion  r^ould  r«y*#l  that  th*  p*rBon  <■  making  assumption*  aoout  what  tha  an* my  .s 
domg  building  up  .-atscn  •  and  about  what  th*  *n*my  is  intanding  to  us*  against  us  *. 


Seal#  tnfar*nca:  W*  uS«  v*Uf  inf*r#h<*  Shill*  hK#n*v*r  w#  draw  CpnciufiOhS  P*J#d  01  ***SOAS  *n<J  #y*danc#  W* 
mtgnt  5*  using  our  d*duetrv«  rationing  infaranc*  skills  Of  our  mductlv*  rasfomng  mf*r*nc#  skill*  V/*  can  apply 
your  Jn*#r*he«  g<iii|  to  all  sort*  o#  thing*  including  p*ti#fi,  oointpns.  JacU  C0h)*ctur*s  pnntiok**  and  assumptions 
tv*  can  *v*n  apply  *u'  mfaranc*  skills  to  mistakat  If  w*  raaton  to  any  conclusions  ot*«d  on  ih.ng*  tnat  v#  know 
■  r*  mlitaktn  than  ar*  moat  lik*ly  going  to  hav#  roachap  a  faulty  conclusion  #\#n  if  .%*  appliad  your  sktllt  wall 
For  ixanplt  r*  know  that  Chicago  is  m  [ItinOvi  Gut  suppos*  w#  .•■*r*  so  confus*d  that  w«  thought  that  Illinois  n*s 
m  Maxico.  and  not  in  th#  Urvi*d  Stitai  Vr*  might  than  mf#r  tnat  Chicago  is  m  *i*k>co  uoed  us*  of  mf*r#n<#  skills 
but  basad  on  mi*tar£*n  b*l<*fi  -  th*  msult  1#.  as  n*  yould  axpact  not  a  tfu*  slatamant.  it  is  imbotont  to  <*#p 
saparat#  what  ■■*  know  to  o«  tru*  and  >khat  conclusions  w*  inf*r  baiad  on  what  w*  know. 


Seal#  Evaluation:  w*  sr#  using  our  avtluabon  skills  w*¥*n  w*  daerd*  how  strong  or  how  w*ak  a  p*rjon  s  argumant* 
»r*  or  hAan  w*  d*t*mnin#  th*  b*h*j  ability  of  a  giv*rt  it  at  a  man  t.  For  *x*mpl*.  what  do  w*  think  about  th*  d*a  that 
th*  sun  goal  around  lh*  **-th*  W*H.  if  w*  w*m  standing  all  day  in  an  oo*n  f+#td  you  might  obs«rv«  tnat  th*  au* 
ros*  in  th*  vast  and  s*t  in  tn#  w*st  Thu  ^uld  s**m  to  support  tn*  id**  that  th*  tun  govs  around  th*  «*rth  Qn 
th*  oth#r  nsnd  if  w*  know  that  th*  aartn  ■*»  spinning  on  its  axis  and  that  th*  solar  tyiUm  »ndud*f  our  pl»n*t  m 


a 
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D  Previous  Studies 


In  an  attempt  to  find  a  research  project  that  dealt  specifically  with  the  critical- 
thinking  skills  of  expert  performers  in  top- tier  computer  science  professionals 
and  that  compared  their  assessment  scores  with  the  same  scores  of  undergrad¬ 
uate  students  this  researcher  reviewed,  among  others: 

•  “Knowledge  Organization  and  Skill  Difference  in  Computer  Program¬ 
mers”  proposed  that  “expert  computer  programmers  can  recall  at  a  glance 
far  more  information  relevant  to  their  field  than  novices  can.”  This  is 
related  to  the  experts’  ability  to  chunk  information  [McKeithen  et  al. 
1981]. 

•  “Empirical  Studies  of  Programming  Knowledge”  asserted  “expert  pro¬ 
grammers  have  and  use  2  types  of  programming  knowledge:  1 )  program¬ 
ming  plans  which  are  generic  program  fragments  that  represent,  stereo¬ 
typic  action  sequences  in  programming,  and  2)  rules  of  programming 
discourse,  which  capture  the  conventions  in  programming  and  govern 
the  composition  of  the  plans  into  programs”  [Soloway  and  Ehrlich  1984]. 
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•  “Sonic  Determinants  of  Skilled  Performance  in  Progranmiing"  proposed 
that  novices  are  not  as  able  as  experts  are  at  “organizing  their  knowledge 
around  semantics,  even  when  dealing  with  the  simplest  code”  [Weiden- 
beck  1984]. 

•  “The  Role  of  Domain  Experience  in  Software  Design”  explored  “the  soft¬ 
ware  designer’s  underlying  constellation  of  knowledge  and  skills,  and  at 
the  way  in  which  this  constellation  is  dependent  upon  experience  in  a 
domain”  [Adelson  and  Soloway  1985]. 

•  “Learning  Flow  of  Control:  Iterative  and  Recursive  Procedures”  studied 
the  mental  models  of  students  and  schema  used  in  programming  and 
found  that  “novices  have  poor  mental  models”  [Kessler  and  Anderson 
1986]. 

•  “Critical  Thinking  Ability  of  Novice  and  Expert  Computer  Program¬ 
mers”  examined  the  critical-thinking  skills  of  college  freshmen  and  seniors 
using  the  Cornell  Critical  Thinking  Test  and  found  that  “(a)  there  is  a 
significant  difference  in  the  critical-thinking  abilities  of  novice  (freshmen) 
and  expert  (senior)  programmers  and  (b)  computer  ability  accounted  for 
more  variation  in  critical  thinking  than  mathematical  ability”  [Hanson 
1986], 

•  “Differences  in  the  Structure  of  Semantic  Knowledge  for  Computer  Pro¬ 
grammers  of  Different  Levels  of  Skill”  “investigated  differences  in  the 
structure  of  semantic  knowledge  for  computer  programmers  of  different 
levels  of  skill  at  the  undergraduate  level”  [Bateson  1987]. 

•  “Programming  in  BASIC  or  LOGO:  Effects  on  Critical  Thinking  Skills” 
studied  “whether  learning  to  program  computers  in  either  the  BASIC 
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or  LOGO  languages  affected  critical-thinking  skills  in  students. . .  (and) 
indicated  that  critical- thinking  skills  were  not  affected  by  age,  gender,  or 
which  computer  language  was  taught  in  an  introductory  course”  [Sat tier 
1987]. 

•  “Problem  Decomposition  By  Computer  Programmers”  studied  the  “roles 
played  by  knowledge  of  task,  content,  and  decomposition  in  the  move¬ 
ment  from  problem  definition  through  solution  design  to  solution  im¬ 
plementation  in  computer  programming. . .  (and)  that  schematic  plans 
, . .  are  inadequate  for  solving  complex  design  problems  and  suggested 
that  a  combination  of  breadth- first,  depth- first  decomposition  is  used 
by  programmers."  The  study  used  a  think-aloud  research  methodology 
to  explore  the  problem-solving  procedures  of  one  expert  and  thirteen 
novice  programmers  [Gong  1988]. 

•  “What.  Best.  Predicts  Computer  Proficiency'.'”’  provided  a  summary  of 
previous  research  into  expert  versus  novice  programming  ability  from 
1970  to  1989  [Evans  and  Simkin  1989]. 

•  “Cognitive  Consequences  of  Programming  Instruction”  studied  “program¬ 
ming  and  how  precollege  programming  instruction  affected  thinking” 
[Linn  and  Dalbey  1989]. 

•  “Effect  of  Computer  Programming  Instruction  on  the  Problem-solving 
Ability  of  College  Level  Introductory  Computer  Students”  explored  the 
“possible  relationship  between  computer  programming  instruction  and 
increased  general  problem-solving  ability.”  This  study  used  a  pre-test  and 
post-test  of  college  freshmen  using  the  “Watson- Glaser  Critical  Think¬ 
ing  Appraisal”  and  found  “no  significant  difference  in  general  problem- 
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solving  ability”  after  one  introductory  computer  science  course  [VanLen- 
gen  1989]. 

•  “The  Effect  of  Computer  Science  Instruction  on  Critical  Thinking  Skills 
and  Mental  Alertness”  surveyed  the  “effect  of  completion  of  an  introduc¬ 
tory  level  computer  programming  course  on  students’  critical-thinking 
and  problem-solving  skills”  [Norris  and  Jackson  1992]. 

•  “Quantitative  and  Qualitative  Differences  Between  Experts  and  Novices 
in  Chunking  Computer  Software  Knowledge”  analyzed  “quantitative  and 
qualitative  differences  between  experts  and  novices  in  knowledge  struc¬ 
ture  and  in  their  chunking  of  computer  software  knowledge”  [Ye  and 
Sal  vend  y  1994]. 

•  “Computer  Programming  and  Analogical  Reasoning:  An  Exploratory 
Study”  probed  the  “possibility  of  designing  (an  introductory)  computer 
science  course. . .  to  include  more  than  just  'coverage’  of  subject  mat¬ 
ter  but  also  to  encompass  questions  of  ‘methods  and  processes’. . .  (of) 
inquiry  (and)  abstract  logical  thinking  that  are  at  the  heart  of  the  intel¬ 
lectual  process”  [Schlafmitz  1996]. 

•  “Differences  Between  Novice  and  Expert  Systems  Analysts:  What  Do 
We  Know  and  What  Do  We  Do?”  identified  “specific  weaknesses  that  set 
novice  and  expert  analysts  apart. . .  (and  suggested)  techniques  that  may 
be  used  to  strengthen  novice  skills.  This  research  supports  the  current 
literature  on  creativity  techniques  as  a  strategy  for  strengthening  system 
analysis  skills”  [Schenk  et  al.  1998]. 

•  “Object-Oriented  Program  Comprehension:  Effect  of  Expertise,  Task 
and  Phase”  discovered  a  “four-way  interaction  of  expertise,  phase,  task 
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and  type  of  model. . .  (and)  show(ed)  that  novices  do  not,  spontaneously 
construct  a  strong  situation  model  but  are  able  to  do  so  if  the  task 
demands  it”  [Burkhard  et  al.  2002]. 

•  “Operationalzing  Predictive  Factors  of  Success  for  Entry  Level  Students 
in  Computer  Science”  examined  the  predictive  value  of  the  Clemson  math 
placement,  examination  on  the  success  of  students  in  a  first-year  computer 
science  course  [Weaver  2004], 

•  “Information  Problem  Solving  By  Experts  and  Novices:  Analysis  of  a 
Complex  Cognitive  Skill”  determined  that  “experts  spend  more  time  on 
defining  problems  and  more  often  activate  their  prior  knowledge,  elabo¬ 
rate  on  the  content,  and  regulate  their  process”  than  novices  do  [Brand- 
Gruwel  et  al.  2005]. 

•  “Pair  Programming  Productivity:  Novice  vs.  Expert”  suggested  that 
“novice  pairs  against  novice  solos  are  much  more  productive  in  program¬ 
ming  performance  than  expert  pairs  against  expert  solos”  [Lui  and  Chan 
2006]. 

•  “Using  Student  Performance  Predictions  in  Computer  Science  Curricu¬ 
lum”  sought  to  develop  a  model  of  success  using  previous  course  grades 
as  a  predictor  of  success  in  future  computer  science  courses  [Chamillard 
2006]. 

•  “Predictors  of  Success  in  a  First  Programming  Course”  used  several  fac¬ 
tors,  including  paper-folding,  map  sketching,  and  phone-book  searching 
to  predict  student  success  in  introductory  computer  science  courses  [Si¬ 
mon  et  al.  2006], 
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•  “What,  Makes  A  Good  Programmer?”  asserted  that  a  person  with  high 
theoretical  value  beliefs,  the  person  who  values  “order,  problem  solutions 
and  proofs,  and  is  motivated  by  the  discovery  of  truth”  plays  a  larger  role 
in  determining  programming  ability  at  the  undergraduate  level  than  does 
cognitive  abilities  or  personality  [Cegielski  and  Hall  2006]. 
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E  Undergraduate  Schools  of  Participants 


•  Alabama  A&M  University 

•  Arizona  State  University 

•  Auburn  University 

•  Brown  University 

•  California  State  University 

•  Carson-Newinan  College 

•  Clemson  University 

•  Coastal  Carolina  University 

•  Cornell  University 

•  Dickinson  College 

•  Drexel  University 

•  East  Tennessee  State  University 

•  Excelsior  College 

•  Florida  A&M  University 

•  Florida  State  University 

•  George  Mason  University 

•  Georgia  Institute  of  Tech 

•  Harvard 

•  Hawaii  Pacific  University 
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•  Indiana  University 


•  Indiana  University  of  Pennslyvania 

•  Jackson  State  University 

•  John  Brown  University 

•  Michigan  State  university 

•  Middle  Tennessee  State  University 

•  Mississippi  Valley  State 

•  Mumbai  University 

•  North  Carolina  State  University 

•  Norwich  University 

•  Notre  Dame 

•  Princeton 

•  Rensselaer  Polytechnic  Institute 

•  Siena  College 

•  SUNY  Geneseo 

•  Texas  A&M  -  Kingsville 

•  The  University  of  Arizona 

•  Transylvania  University 

•  TVoy  University 

•  Tulane  University 
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•  United  States  Military  Academy 


•  Univ  of  Southern  Mississippi 

•  University  of  Puerto  Rico 

•  University  of  Akron 

•  University  of  Alabama 

•  University  of  Delaware 

•  University  of  Florida 

•  University  of  Iowa 

•  University  of  Nebraska 

•  University  of  Northern  Colorado 

•  University  of  Southern  California 

•  University  of  Texas- Arlington 

•  University  of  Texas-San  Antonio 

•  University  of  Virginia 

•  University  of  Wisconsin-Green  Bay 

•  United  States  Air  Force  Academy 

•  VJTI.  Mumbai 

•  Weber  State  University 

•  Western  Oregon  University 
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F  Military  Participants 
F.l  DISA 

For  more  information  on  DISA  see,  www.disa.inil. 


Vhtrtin, 

VtlUM 

Core  ntuion 
Are« 

DISA  MrtlrtV 

DISA  Hiao#V 

OrgAni/tition 

Structure 

I  nrcjurnltv  Aikerf 
Qufiirtiaei 

DISA  A  -  * 


Mission,  Vision,  and  Values 


Mission: 


me  Defense  information  Systems  Agency  is  a  combat  support  agency  responsible 
for  planning,  engineering,  acquiring,  fielding,  and  supporting  global  net-centnc 
solutions  to  serve  the  needs  of  the  President,  Vice  President,  the  Secretary  of 
Oefense,  and  other  DoD  Components,  under  all  condition*  of  peace  and  war. 


Vision: 


We  are  the  provider  of  global  net-centnc  solutions  for  the  Matron  s  warfighters  and 
alt  those  who  support  them  in  the  defense  of  the  nation. 


Values: 


The  people  of  DtSA  are  committed  to: 


*  Publication  t 
,  rnrpoMlf 

Comm  urue  eUoo* 

-  bOIA 

-  NO  1 1 A  H  uete 


•  Guarantee  out  forces  global  information  dominance  by  providing  jointly 
interoperable  systems,  assured  security:  survivabdity,  availability;  superior 
quality. 

•  The  best  innovative  ideas,  excellence  m  design  and  engineering  speed  and 
agiht>  m  execution,  and  the  best  value  integrated  information  solutions  for  the 
DoD. 

•  Active  listening,  active  partnering,  operational  and  individual  accountability  ■ 
consistently  exceeding  our  customers  expectations 

•  Each  other  and  a  common  purpose,  in  an  environment  of  change,  through 
bonds  of  integrity,  trust,  support,  and  teamwork. 
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Defense  Information  Systems  Agency 

Depart  merit  of  Defense 


W  Hair 
Contact  l* 
5t*n;Fi 
U1SA  A  jj 


Mom*  My  DISA 


Pr«**t  ftetourcet  Services  ft  Solution*  Customer  Support  Joh&ODI&A 


Values 

p  Coro  H»tiun 
Ar«Ai 

-  DlSA  Str«t(!i)v 

•  DISA  HiMary 

a  Urg>inir.ili4m 
Structure 

t  Frequently  Asfced 

Qur^tipn^ 

•  D1SA  A  I 

•  Ncwi  A  events 
■  Putt  Ik.  ot  ions 


Cor*  Nfiuofl  Artat 

D1SA  performs  a  number  of  very  important 
mis sion 5  m  support  of  the  President,  the 
Secretary  of  Defense,  the  Joint  Chefs  of  Staff,  the 
Combatant  Commanders,  and  the  other 
Department  of  Defense  <DoO)  components  under 
ail  conditions  of  peace  and  war.  Some  of  these 
missions  are  designated  as  core  missions  because 
together  they  provide  highly  integrated  C4 
warfighting  capabilities.  The  whole  is  greater  than 
the  sum  of  the  parts  and  removing  one  of  the  core 
missions  would  adversely  affect  the  others.  Some 
other  critical  missions  are  designated  as  best  fit  missions,  meaning  that  PISA  is 
well -suited  to  perform  these  missions  and  has  been  assigned  them  over  time. 
However,  they  could  be  assigned  to  others  without  destroying  the  synergy  that 
exists  among  the  core  missions.  Thus,  the  terms  core  and  best  fit  are  not 
designators  of  relative  importance  but  indicate  degrees  of  synergy  The  designated 
core  missions  of  PISA  are  communications,  joint  command  and  control,  defensive 
information  operations,  combat  support  computing,  and  joint  interoperabilit  y 
support. 


wernfeatSvaaert 

Co-nnutirna 

JJiuranca 

Jaot  Co'-nm ir>o  ltd 

JlvM  ['tUwrsbilihr 

S  l,  apart 


Corporate 


F.2  Signal  Corps 

For  more  information  see  the  School  of  Information  Technology  website, 
littp:/ / ww  w .  gor  don  .army.mil/sit/ . 


U.S.  Army  Signal  Center  MRsi 
Fort  Gordon 


Public  Affairs  Office 


AKQ 


DOtM 

Tnabb  Tick*! 


ttrtiiirH  I  Ijjrm**  I  .trap  flumr  I'mp  I  Stmt 
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LticUiny  t tvi  way  in  Todays  ieuuwioyy ' 


Vy 

L 


&  Otter  Lii^cs  \  General  3 


OL  R  MISSION 

To  prov  ide  trained  Infoimation  Technology  QT)  soldiers  and  Chilians  to  the  Anny  and  the 
joint  community  The  resident  training  is  done  through  classroom  instruction,  hands-on 
practical  exercises,  and  interaction  with  simulation  equipment  for  critical  IT  skills  associated 
with  networking  concepts  and  colourations,  network  management,  systems  administration 
and  securin’,  and  information  dissemination  SIT  conducts  both  resident  and  non-resident 
functional  courses  in  support  of  Information  Assurance  (LA)t  Communications  Securin' 
(COMSEC)-  Defense  Message  Systems  (DMS),  and  Joint  Networks  The  SIT  trains  all  IT 
officers  in  die  grades  warrant  officer  one  through  lieutenant  colonel  in  officer  Areas  of 
Concentration  (AOC)  within  the  Signal  Regiment's  spectrum  at  die  basic  and  advanced 
levels  These  AOCs  include  Functional  Area  (FA)  Officers  53 A  and  24A,  AOC  25A.  and, 
250X,  251  A,  and  254A  Signal  Corps  Officers  All  Signal  Corps  noncoramissiofied  officers 
(NCOs)  attend  portions  of  FT  courses  at  both  the  basic  and  ach'anced  levels  Many  of  the  IT 
students  are  DoD  Chilians  as  well  as  Allied  Students  who  attend  under  the  International 
Mifitary  Student  Program  conducted  by  the  Department  of  State 

Tc:  cotanwan  or  CONCERNING  THE  VESSHE  comict  Vfcb  ^dmimslrator 


mi 
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G  Assessment  Procedures 


G.l  Facility  Recruitment  Letter 

I  am  a  graduate  student  working  under  the  direction  of  Dr.  Steve  Stevenson 
and  am  conducting  research  on  traits  of  creativity  and  critical- thinking  skills 
of  computer  science  experts  and  computer  science  students. 

I  am  seeking  volunteers  to  take  a  45-minute,  computerized  assessment  of 
critical-thinking  abilities,  “The  California  Critical  Tliinking  Skills  Assessment, 
’’produced  by  Insight  Assessments.  It  is  a  standardized  validated  assessment 
that  will  provide  you  with  immediate  results. 

I  will  use  the  faculty  scores  as  my  “control”  expert  group  and  as  a  com¬ 
parison  with  student  scores.  No  identifying  information  will  be  collected  from 
faculty  members  with  their  scores.  I  will  collect  educational  data,  including 
GPA,  HS  CPA,  and  SAT  scores  from  the  students  who  take  these  same  assess¬ 
ments  to  see  if  there  is  any  correlation  between  test  results. 

Please  respond  to  this  email  (deanb@clemson.edu)  to  schedule  a  time  to 
take  the  test.  Thank  you  in  advance  for  your  help! 

Respectfully, 

Dean  Bushey 

G.2  Assessment  Instructions 

Volunteers  received  the  following  survey  instruction  letter: 

Thank  you  for  volunteering  to  help  me  in  my  research  by  taking  the  Cali¬ 
fornia  Critical  Thinking  Skills  Test.  Allow  40  to  45  minutes  for  the  test. 

Steps  to  follow: 

1.  Go  to:  http://www.insightassessment.com/ia/login.asp 
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2.  Login  id:  xxxxxxxx 
Password:  xxxxxxxx 

If  you  have  problems  with  this  username/password,  let  me  know  and  I 
will  give  you  an  alternate 

3.  Click  the  tab  at  the  top  (4th  from  left)  entitled  “Start  Test/Survey.” 

4.  You  will  see  the  on-line  Testing/Survey  Tool  Page. 

5.  Click  the  “START  ONLINE  TESTING /SURVEY”  in  the  top  paragraph. 
You  may  need  to  install  a  Java  web  application.  It  should  go  smoothly, 
but  email  me  if  you  have  questions. 

The  preceding  steps  should  launch  the  test.  You  will  have  to  re- login 
with  the  same  user  id  and  password  as  above. 

6.  Try  it  out  -  If  you  have  any  problems,  please  call  or  email  me. 

Thank  you  again  for  your  participation. 
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Institutional  Board  Review  Approval 


CLEMSON 

UNIVERSITY 

January  22,  2007 

Dr.  Steve  Stevenson 
Computer  Science 
315  McAdams  Hall 
Clrmson  University 
Clemaon,  SC  29634 

SUBJECT  Human  Subjects  Proposals  06-IRB-226  entitled  “Creativity  and  Critical 
Thinking  Skilli  of  Computer  Science  Students". 

Dear  Dr,  Stevenson: 

The  Institutional  Review  Board  (IRB)  of  Clcmson  University  reviewed  the  above- 
mentioned  study  using  Expedited  review  procedures  and  haft  recommended  approval. 

Approval  for  this  itudy  hai  been  granted  u  or  January  22,  2007. 

Your  approval  period  is  January  22,  2007  to  January  21,  2008.  Your  next  continuing 
review  is  scheduled  for  November  2007  Please  refer  to  the  IRB  number  and  title  in 
communication  regarding  this  study,  Attached  is  handout  regarding  ihe  Principal  and 
Co-In  vestigatore’  re&ponsibi lilies  in  the  conduct  of  human  research.  The  Co-Investigator 
responsibility  handout  should  be  distributed  to  all  members  of  the  research  team. 

No  change  in  this  approved  research  protocol  can  be  initiated  without  the  IRB’s  approval. 
This  includes  any  proposed  revisions  or  amendments  to  the  protocol  or  consent  form. 

Any  unanticipated  problems  involving  risk  to  subjects,  any  complications,  and/or  any 
adverse  events  must  be  reported  to  the  Office  of  Research  Compliance  immediately. 
Please  contact  the  office  if  your  study  has  terminated  or  been  completed  before  the 
identified  review  date. 

We  appreciate  your  assistance  m  complying  with  federal  regulations  and  institutional 
policies.  Yon  may  contact  the  Office  of  Research  Compliance  at  656-6460  if  you  have 
any  questions. 


© 

OFFICE  OF  HESEAJICH  COMPLIANCE 
JHA  HtJI  BmMSTW  CkrwSCMU+^TM  164*5*  LS»  FAXWUdW-WS  m 

liyntuiKKial  Hmt*  Bant  W4d56.d4dO  I (MW™I  Sta»fc*v  864^^.01 18  Ar,.f«i  Rwafth  Cum Almr  864.d5fi.4H8 
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I  Supplemental  Data 


Raw  data  collected  for  this  research  is  listed  in  Table  I.  Individual  identifying 
information  has  been  removed  as  directed  by  the  Clemson  IRB. 
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ID  Race  Gender  Age  Ed  level  Maj  V  SAT  M  SAT  CUM  HRS  GPR  Ind  Ded  Ana  Inf  Eva)  TOT 

31787  Caucasian  Male  41  PhD  Computer  Science  0  0  0  0  14  12  7  11  8  26 

31942  Caucasian  Male  23  Senior  Computer  Science  610  650  137  1.97  12  16  5  14  9  28 
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32968  Caucasian  Male  63  PhD  Industrial  Engineering 

32969  Caucasian  Male  30  Masters  Computer  Science 

32970  Caucasian  Male  28  Masters  Computer  Science 

Ganlmued  on  next  page 


[D _ Race  Gender  Age  Ed  level _ _ Major  V  SAT  M  SAT  CUM  HRS  GFR  Ind  Ded  Ana  Inf  Eva!  TQT 

32973  Caucasian  Male  50  PhD  Computer  Science  0  0  0  0  14  14  6  15  7  28 

32976  Caucasian  Male  42  Masters  Engineering  Physics  0  0  0  017  13  712  11  30 

32977  Black  Male  32  Masters  Industrial  Engineering  0  0  0  0  15  15  714  930 
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33165  Caucasian  Male  19  Freshman  Civil  Engineering 

33166  Caucasian  Male  18  Freshman  systems  engineering 

33167  Caucasian  Female  18  Freshman  Undecided 

33168  Black  Male  19  Freshman  Foreign  Area  Studies 

Conitnued  on  next  page 


ID _ Race'  Gender  Age  Ed  level _ Major  V  SAT  M  SAT  CUM  HRS  GPR  Ind  Ded  Ana  Inf  Eval  TOT 

33169  Caucasian  Male  18  FYeshman  Undeclared  0  0  0  012  9  5  10  6  21 

33170  Caucasian  Male  21  Freshman  Behavioral  Science  0  0  0  0  15  15  6  13  11  30 

33171  Caucasian  Male  22  Freshman  Undeclared  0  0  0  0  12  4  3  8  5  16 
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35915  Caucasian  Male  35  Masters  Economics 

35918  Black  Female  33  Bachelors  CIS 

35919  Caucasian  Male  35  Senior  Criminal  Justice 

35921  Black  Male  39  Bachelors  Political  Science 

Continued  on  next  page 


[D  Race  Gender  Age  Ed  level  Major  V  SAT  M  SAT  CUM  HRS  GPR  Ind  Ded  Ana  Inf  Eval  TOT 

35923  Black  Male  38  Senior  Criminal  Justice  0  0  0  011  6  56  6  17 

35924  No  Data  Male  30  Senior  General  Studies  0  0  0  09435513 

35925  Caucasian  MaJe  32  Bachelors  sociology  0  0  0089674  17 
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36594  Hispanic  Male  42  Bachelors  CIS 

36611  Caucasian  Male  37  Masters  Management 

36614  Caucasian  Female  45  Bachelors  CSC 

36615  Caucasian  Male  41  Bachelors  Electrical  Engineering 

Continued  on  next  page 


ID _ Race  Gender  Age  Ed  level _ Major  V  SAT  M  SAT  CUM  HRS  GPR  Itid  Ded  Ana  Inf  Evai  TOT 

36666  Caucasian  Female  33  PhD  Systems  Engineering  0  0  0  0  15  16  7  15  9  31 

36795  Caucasian  Female  51  Masters  Chemistry  0  0  0  0  14  12  711  8  26 

36797  Caucasian  Female  21  Senior  Mathematical  Sciences  630  800  172  3.88  15  13  7  12  9  28 
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Insight  Assessment 

CCTST  2000  Interpretation  Document 


This  document  provides  score  interpretation 
information  for  the  CCTST.  Your  Capscore™  results 
will  include  tesMakers  scores  and  corresponding 
percentile  scores  that  are  based  on  the  data 
provided  in  this  document. 

The  aspects  of  critical  thinking  measured  by  the  CCTST 
are  defined  below.  Comparison  norms  follow, 

CRITICAL  THINKING  DEFINED 

The  CCTST  is  based  on  the  APA  Delphi  consensus 
conceptualization  of  Critical  Thinking*  described  in  the 
following  section  of  this  manual.  This  conceptualization 
of  CT  is  an  historically  important  benchmark.  It  is  an 
expression  of  expert  consensus  articulated  without  the 
constraints  of  accreditation  or  legislation,  and  based  on 
the  participation  of  46  leading  theorists,  teachers,  and 
CT  assessment  specialists  from  several  disciplines. 
This  conceptualization  of  CT  was  reaffirmed  in  the 
1993/1994  national  survey  and  replication  study 
conducted  by  the  National  Center  for  Higher  Education 
Teaching,  Learning  and  Assessment  at  The 
Pennsylvania  State  University.1 

These  experts  characterize  critical  thinking  as  the 
process  of  purposeful,  self- regulatory  judgment/*  Critical 
thinking,  so  defined,  is  the  cognitive  engine  which  drives 
problem-sofving  and  decision-making.  This  robust 
concept  of  CT  supplied  the  conceptual  architecture  used 
to  address  the  US  Department  of  Educations  Education 
Goals:  2000  mandate4  In  that  context  it  became  the 


American  Philosophical  Association  (1990)  Critical  Thinking:  A 
Statement  ...sf  Expert  Consensus  for  Purposes  of  Educational 

Assessment  and  InsiMioa.  ('The  Delphi  Report')  ERl£  Ooo  No  ED 
31&423,  pp.  80.  An  executive  summary  including  tables  and 
recommends! ions  (pp.  22)  also  available-  through  The  California 
Academic  Press 

: Jones  EA.  Hoffman  5,  Moore  LM.  Ratdrfl  G.  Tibbetts  S,  Click  BL 
National  Assessment  of  College  Student  learning.  Identifying  the 
Collage  Graduate  Essential  Shifts  in  Writing,  Speech  end  Listeningr 
and  Critical  Thinking.  Washington  DC:  National  Center  for  Ed  national 
Statistics  US  Department  of  Education.  Office  of  Educational 
Research  and  Improvement,  1995.  DERI  publication  NCES  93-001 . 

',The  American  Philosophical  Association.  (1990)  Critical  Thinking:  A 
Statement  of  .Expert  Consensus  for  Purposes  of  Educational 

Assessment  and  Inamjction.  ('The  Delphi  ReporT),  ERIC  Doc  No  ED 
315-423,  pp  SO  [Executive  summary  including  tables  and 
recommends! ions  (pp  22)  also  available  through  The  California 
Academic  Press.  21 7  La  Cruz  Ave  r  Millbrae  CA,  94030  ] 

4 National  Center  for  Educational  Statistics.  Commissioned  papers  for 
November  1991  National  Conference.  Office  of  Educational  Research 
and  Improvemenl.  US  Department  of  Education.  ERIC  Document 
Numbers  340753  through  340768,  1992. 


framework  of  a  national  replication  study  of  the  definition 
and  valuation  of  CT  which  yielded  a  broad  consensus 
among  hundreds  of  educators,  employers,  and  policy 
makers  * 

The  skills  of  Analysis,  Evaluation,  and  Inference  are 

specifically  targeted  by  the  CCTST.  These  are 
described  below. 

Analysis  as  used  on  the  CCTST  has  a  dual  meaning 
First  it  means  "to  comprehend  and  express  the  meaning 
or  significance  of  a  wide  variety  of  experiences, 
situations,  data,  events,  judgments,  conventions,  beliefs, 
rules,  procedures  or  criteria,"  which  includes  the 
sub*skills  of  categorization,  decoding  significance,  and 
clarifying  meaning  Analysis  on  the  CCTST  also  means 
"to  identify  the  intended  and  actual  inferential 
relationships  among  statements,  questions,  concepts, 
descriptions  or  other  forms  of  representation  intended  to 
express  beliefs,  judgments,  experiences,  reasons, 
information  or  opinions,"  which  includes  the  sub-skills  of 
examining  ideas,  detecting  arguments,  and  analyzing 
arguments  into  their  component  elements 

Evaluation  as  used  on  the  CCTST  has  a  dual  meaning. 
First  it  means  "to  assess  the  credibility  of  statements  or 
other  representations  which  are  accounts  or  descriptions 
of  a  person's  perception,  experience,  situation, 
judgment,  belief  or  opinion;  and  to  assess  the  logical 
strength  of  the  actual  or  intended  inferential  relationships 
among  statements,  descriptions,  questions,  or  other 
forms  of  representations,"  which  includes  the  sut^skills 
of  assessing  claims  and  assessing  arguments 
Evaluation  on  the  CCTST  also  means  "to  state  the 
results  of  one's  reasoning;  to  justify  that  reasoning  in 
terms  of  the  evidential,  conceptual,  methodological, 
criteriological  and  contextual  considerations  upon  which 
one's  results  were  based;  and  to  present  one's 
reasoning  in  the  form  of  cogent  arguments*  which 
includes  the  sub-skills  of  stating  results,  justifying 
procedures,  and  presenting  arguments. 


*  Jonas  E,,  Cofrallo  S.,  Faoone,  P .  4  Ratcliff  G ,  Developing 
consensus-  for  Critical  thinking.  Paper  presented  at  the  annual 
meetings  of  the  American  Association  of  Higher  Education, 
Washington  DC.  June  1994  (This  work  was  also  presented  at  the 
Sixth  International  Conference  on  Thinking,  MIT.  Boston.  MA,  July 
1994  and  at  The  Fourteenth  International  Conference  on  Critical 
Thinking,  Songm®.  CA.  August  1994.)  This  survey  research  was 
based  on  the  "Critical  Thinking  Goals  Inventory,"  developed  by  Beth 
Jones  and  Gary  Ratcliff  (US  Department  ot  Education.  Office  ot 
Educational  Research  and  Improvement  Grant  R1 1760037,  CFDA  No 
84.1 17G).  For  additional  information  contact  (he  National  Center  tor 
Higher  Education  Teaching.  Learning  &  Assessment.  The 
Pennsylvania  State  University,  403  South  Allen  Street.  Suita  104, 
University  Park.  PA  1 6801 
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Inference  as  used  On  the  CCTST  moans  "to  identify  and 
secure  elements  needed  to  draw  reasonable 
conclusions;  to  form  conjectures  and  hypotheses,  to 
consider  relevant  information  and  to  educe  the 
consequences  flowing  from  data,  statements,  principles, 
evidence,  judgments,  beliefs,  opinions,  concepts, 
descriptions,  questions,  or  olher  forms  of 
representation,"  which  includes  the  sub-skills  of 
querying  evidence,  conjecturing  alternatives,  and 
drawing  conclusions. 


The  following  traditional  scores  are  also  provided 

Deductive  Reasoning  as  used  in  the  CCTST  sub-scale 
means  the  assumed  truth  of  the  premises  purportedly 
necessitates  the  truth  of  conclusion 
Inductive  Reasoning  as  used  in  the  CCTST  sub- scale 
means  an  argument’s  conclusion  is  purportedly 
warranted,  but  not  necessitated,  by  the  assumed  truth  of 
its  premises.  Scientific  confirmation  and  experimental 
disconfirmation  are  examples  of  inductive  reasoning 

The  Inductive  and  deductive  scales  overlap  with  the 
analysis,  inference,  and  evaluation  scales  analysis, 
inference,  and  evaluation  add  up  to  the  CCTST  total 
score.  Induction  and  deduction  also  add  up  to  the 
CCTST  total  score 


Norm  Sample 


This  is  an  aggregated  sample  of  4-year  college  students. 
Descriptive  Statistics: 


Variable 

N 

Mean 

Median 

TrMean 

StDev 

SE  Mean 

Total 

2677 

16 . aoi 

16*000 

16*729 

5.062 

0*098 

Analysis 

2677 

4,1178 

5.OO0O 

4,4645 

1,4080 

0.0272 

Inference 

2677 

7*8450 

8.0000 

7,8144 

2,6848 

0.0519 

Evaluation 

2677 

4*5185 

4.0000 

4 , 4  612 

2,1431 

0,0414 

induction 

2677 

0,5293 

10.0000 

9.5509 

2.B217 

0,0545 

deduction 

2677 

7,2719 

7 . 0000 

7,1876 

2.8897 

0,0559 

Variable 

Minimum 

Maximum 

01 

Q3 

Total 

1  *  0000 

12,0000 

13.0000 

20,0000 

Analysis 

0.0000 

7.0000 

4.0000 

5.OOQ0 

Inf erence 

0.0000 

15.0000 

6.0000 

10.0000 

Evaluation 

0.0000 

11,0000 

3.0000 

6.0000 

induction 

0.0000 

17.0000 

8.0000 

11,0000 

deduction 

0.0000 

16*0000 

5.0OOO 

9*0000 
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Evaluation 

evaipct 

0 

0.005977 

1 

0036048 

2 

0.119724 

3 

0.263915 

4 

0.439484 

5 

0  609451 

6 

0,754763 

7 

0.864027 

8 

0.931453 

9 

0969369 

10 

0.989354 

11 

0,997945 

Inference 

infpet 

0 

0.000187 

1 

0.002615 

2 

0.010066 

3 

0.029697 

4 

0.071722 

5 

0.146806 

6 

0.257191 

7 

0.397633 

8 

0.544453 

9 

0.675196 

10 

0.781285 

11 

0.864774 

12 

0  92585 

13 

0.963205 

14 

0.986739 

15 

0.997385 

16 

1 

Analysis 

ana  pet 

0 

0.002241 

1 

0.015129 

2 

0.058461 

3 

0.166604 

4 

0.365708 

5 

0.628129 

6 

0,854688 

7 

0.971236 

Total 

Totpct 

0 

q 

1 

0.000187 

2 

0.00056 

3 

0.001121 

4 

0.002428 

5 

0.004109 

6 

0.008218 

7 

0.018491 

6 

0.033433 

9 

0.052858 

10 

0.080874 

11 

0.120844 

12 

0.175196 

13 

0,237393 

14 

0.305379 

15 

0.382144 

16 

0.461898 

17 

0.541091 

18 

0.613934 

19 

0.676317 

20 

0.737206 

21 

0  793052 

22 

0.839746 

23 

0,879716 

24 

0.910534 

25 

0.934442 

26 

0,954053 

27 

0969742 

28 

0,980388 

29 

0.98898 

30 

0.995704 

31 

0,998879 

32 

0.999813 

33 

1 

34 

1 
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Induction 

in  d  pet 

0 

0.00056 

1 

0.001307 

2 

0.002988 

3 

0.01102 

4 

0.02951 1 

5 

0.061076 

3 

0.110758 

7 

0.190325 

8 

0.299589 

9 

0.422488 

iq 

0.555286 

ii 

0.690325 

12 

0.B01457 

IS 

0,883638 

14 

0.941726 

15 

0.975906 

16 

0.993463 

17 

0.999253 

Deduction 

dedpet 

0 

0.000747 

1 

0.003736 

2 

0.016063 

3 

0.055286 

4 

0.127195 

5 

0.220614 

6 

0.361972 

7 

0.502055 

8 

0.620315 

9 

0.73014 

10 

0.019201 

11 

0.879903 

12 

0.928838 

13 

0.962832 

14 

0.983377 

15 

0.993463 

16 

0.998319 

17 

1 
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